2021年9月3日 星期五

[backend, main] The NodeJS, Express/Koa, Web FAQ

The NodeJS, Express/Koa, Web FAQ

井民全, Jing, mqjing@gmail.com



The Public FAQ Index (view)


Table of contents


1. Mindset 3

1.1. RWD 3

2. Tools 3

2.1. Editor 3

2.2. Package manager 3

2.2.1. yarn 3

2.2.2. npm 5

2.3. Test 6

2.3.1. Unitest 6

2.3.2. Integration test 7

2.3.3. Report 8

2.4. Bundle 8

2.4.1. Webpack 8

2.4.2. Gulp 10

3. Node 10

3.1. Mindset 10

3.2. Test 10

3.3. Debug 10

3.4. Setup 10

3.5. Docker 12

3.6. Watch 13

3.7. Buffer 13

3.8. Process 14

4. Libraries 15

4.1. JSON 15

4.2. CRUD 15

4.3. Asynchronous 16

4.3.1. Promise 16

4.3.2. Stream, RxJS 16

4.4. Data structure 18

4.5. Network 18

4.5.1. Mqtt 18

4.5.1.1. Plan Mqtt 19

4.5.1.2. Mqtt over Websocket 19

4.5.2. Http server 19

4.5.3. Net 20

4.5.4. Socket.io 21

4.6. HTTP Client 23

4.7. Html Parser 28

4.7.1. JQuery 28

4.7.2. htmlparser2 28

4.7.3. cheerio 29

4.8. Bluetooth 29

5. Backend 30

5.1. Security 30

5.2. Web Server 30

5.3. Database 30

5.3.1. Comm 31

5.3.2. Mongodb 31

5.4. Flask 31

5.5. KOA/Express 31

5.6. Admin (User) Management 34

5.7. Auth: for User Signup & User Login 34

6. Hybrid App 34

7. Others 35



1. Mindset

  1. Async stream 是以 byte 為單位, 所以中斷的部份也是以 byte 為主

1.1. RWD

  1. [book] HTML5與CSS3響應式網頁設計 第二版


2. Tools

2.1. Editor

  1. [stackBlitz] On-line editor that you can see the results in real time (ref)

2.2. Package manager

2.2.1. yarn

  1. [yarn, npm] npm and yarn translation (ref)

  2. [yarn, install] How to install yarn (view)


curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -


echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list


# (A) for normal node installation

sudo apt-get install yarn    


# (B) for nvm node installation

sudo apt update && sudo apt install --no-install-recommends yarn


sudo npm install -g yarn


  1. [yarn, version] How to check module version


yarn info $module-name version   


yarn why $module


  1. [yarn, run] How to run yarn from any folder? (view)


yarn --cwd ../../../myprj/myapp/ install

yarn --cwd ../../../myprj/myapp/ start

  1. [yarn, script, npm-run-all] How to run multiple npm-scripts in parallel or sequential (view)


Install

yarn add npm-run-all -D


Run all

"build:dev": "npm-run-all script1 script2"


  1. [yarn, script, makeclean, cross-platform] A npm script for clean


Install

yarn global add rimraf // yarn add rimraf -D


clean

"clean": "rimraf out build dist node_modules & rimraf **/*.js & rimraf **/*.map & rimraf **/*.raw & rimraf **/*.log",

  1. [yarn, script, copyfiles, cross-platform] A npm script for copy files (ref)


#copy all ths js files to the build folder

npx copyfiles -u 1 src/*.js build

  1. [yarn, script, invoke, cross-platform] How to invoke a npm script without terminated before the next call, Windows


call yarn [your-npm-script-name]




2.2.2. npm

  1. [npm] What's the different between npm install --savedev <package-name> (ref)


The flag --save-dev will tell NPM to install Typescript as a devDependency. The difference between a devDependency and a dependency is that devDependencies will only be installed when you run npm install, but not when the end-user installs the package.

For example, Typescript is only needed when developing the package, but it’s not needed while using the package.



  1. [npm, install] --save vs. --save-dev (ref)


--save: the package that will be bundled into your production bundle

--save-dev: the package for development purposes (e.g. a linter, testing libraries, etc.)

  1.  

  1. [npm, version] How to check a module version


npm -v $module-name


  1. [npm, module, install] How to install a module (ref)


sudo npm install npm@latest -g     # upgrade the npm

npm init

npm install my_package


  1. [npm, package.json, start] How is npm start?


# Run the command in package.json's start property of its scripts object.


# in file package.json

"scripts": {

     ...

     "start": "node index.js",

     ...

}


  1. [npm, proxy] How to setup npm behind a proxy (view)

  2. [npm, proxy] How to setup npm proxy (view)

  3. [npm, package.json, clean] How to clean the node app 


# add line to package.json

...

"clean": "rm -rf node_modules",

...


# run the clean

npm run clean

  1. [npm package.json, build-clean] clean script in jest


"build-clean": "rimraf './packages/*/build' './packages/*/tsconfig.tsbuildinfo'",

  1. [npm, package.json, clean-all] clean script in jest


"clean-all": "yarn clean-e2e && yarn build-clean && rimraf './packages/*/node_modules' && rimraf './node_modules'",


2.3. Test

  1. [jest, official] Getting Start (ref)

  2. [jest, mindset] Running Jest Tests Before Each Git Commit (ref)

2.3.1. Unitest

  1. [jest, unit-test, react, doc] Create react app typeescript: testing with jest and enzyme (ref)

  2. [jest, unit-test, doc] Typescript deep dive, Jest (ref

  3. [jest, unit-test] How to use Jest to test your typescript project (view)


  1. yarn add jest @types/jest ts-jest typescript -D

  2. vi ./jest.config.ts   // for config jest

  3. vi ./package.json // for yarn test

  4. ./src/${your_typescript_source}       

  5. ./src/__tests__/${unit-test-modules}   // put your unit-test modules

  1. [jest, unit-test, function] How to create an function unit-test


// src/foo.ts

export function myFun(n: number):number {

return n%2;

}


// src/__tests__/foo.test.ts

import {myFun} from '../foo';


test('myfun: even test', () => {

  expect(myFun(2)).toBe(0);

})


test('myfun: odd test', () => {

  expect(myFun(3)).toBe(1);

})

2.3.2. Integration test

  1. [jest, int-test, doc] How to write a integration test for your project using jest (ref)


  1. All integration related codes live in <rootDir>/tests

  2. The general pattern looks like as following


import TestApp from ....;


let testApp;

beforeAll(async () => {

  testApp = new TestApp();

  await testApp.start();

});


afterAll(() => testApp.stop());


test("something", async () => {

  });


  1. [jest, int-test] How to run different jest configuration files for unit-test and integration test (ref)


File: package.json

{

    "test": "jest -c jest.config.js",

    "unit": "jest -c jest.config.unit.js", 

    "Integration": "jest -c jest.config.integration.js"

}



2.3.3. Report

  1. [jest, report] How to get the code coverage with jest (ref)


jest --coverage



2.4. Bundle

Have dead code elimination — that basically means that Webpack can find code that you are not actually using and not add it to the final bundle. Bundle up smaller modules of JavaScript so we are not serving your entire application to every single user that just visits your homepage [ref].

2.4.1. Webpack

  1. [webpack, official] The official site (ref)

  2. [webpack, official] Getting Start (ref)

    1. If a dependency is included but not used, the browser will be forced to download unnecessary code.

  3. [webpack, tutorial] Webpack, what is it and is it better than gulp? (ref)

    1.  看到這段 However, as Uncle Ben said: “with great power comes great responsibilities” 我笑了.

  4. [webpack, tutorial] webpack tutorial gitbook (ref) (original)

    1. focus on being simple, tackle one problem at a time; basic building blocks that add up, step by step, to making you feel like a developer wearing a jetpack! (ref)

  1. 只要設定 webpack.config.js 指定 entry 以及 output 即可運作

  2. Why webpack => uglify, optimized the size


  1. [webpack, uglify] UglifyJsPlugin (ref) (chinese)

  2. [webpack, typescript] How to use Webpack to bundle the typescript webapp*** (view)


# Step 1:  install dependency

tsc --init

yarn init

yarn add -D webpack webpack-cli typescript @types/webpack ts-loader ts-node @types/node


yarn add -D html-webpack-plugin


# Step 2: create webpack.config.ts

# * entry file

# * html-webpack-plugin


# Step 3: bundle

$webpack


Verification

gio open dist/index.html


  1. [webpack, html, template] How to html-webpack-plugin template function for your project (view)

    1. Note: Not necessary to add script to your template, the html-webpack-plugin will insert the script src command when bundle.

// File: webpack.config.ts

...

plugins: [

        new HtmlWebpackPlugin({

            template: 'src/index.html'

          })

    ],


...

  1. [webpack, dev server] How to setup the webpack dev server (view

  2. [webpack, copy] How to copy individual files or entire directories to the build directory using Webpack (view)

2.4.2. Gulp

  1. [gulp, official] The official site (ref)

  2. [gulp, uglify] How to use gulp-uglify (view), (ref)

  3. [gulp, clean] How to do a clean task (view)

  4. [gulp, concate] How to concat a serial js files  (view)

  5. [gulp, clean, bundle, copy] How to use yalp for clean, bundle, copy (view)


3. Node

3.1. Mindset

  1. [import] 必須要在 project root 以下, 不要用 symblic 連到 project 以外.

3.2. Test

  1. [node, test, loadtest] How to use loadtest for your node service (view)

  2. [node, test, process manager] Node.JS Process Manager, Introduction, json web token, (view)

  3. [node, test, unittest, assert] How to use assert in  your node (view)

  4. [node, test, unittest, nodeunit] How to use nodeunit for your node code (view)

3.3. Debug

  1. [debug] How to use node debug, step in (view)

3.4. Setup

  1. [node, embeddy, doc] Nodejs for Embeddy System (ref)

  2. [node, install, nvm] How to install Node.JS via nvm


curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.5/install.sh | bash


export NVM_DIR="$HOME/.nvm"


[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"


nvm install v8.6.0


  1. [node, nvm, uninstall] How to uninstall nvm (view) (ref)


rm -rf ~/.nvm && rm -rf ~/.npm && rm -rf ~/.bower

vi ~/.bashrc

find and remove line source ~/.nvm from yours .bashrc or .zshrc



  1. [node, install] How to install Node.JS via  Node.js version manager, n (ref)


# install n

curl -L https://git.io/n-install | bash


// or automatic install version (without ask)

curl -L https://git.io/n-install | bash -s -- -y  


// usage 

n 12


#uninstall n

$n-uninstall -y


  1. [node, install] How to install Node.JS (view)


#!/bin/bash

sudo apt-get update

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -

sudo apt-get install -y nodejs


  1. [node, version, n] How to downgrad node version (ref) (github)

    1. n is not supported natively on Windows 

#switch node version

$ sudo npm install -g n

$ n 6.10.3

this is very easy to use.


then you can show your node version:


$ node -v

v6.10.3


# install the latest version

sudo n latest

  1. [node, version, nvm] How to change the node version (ref)


curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash


command -v nvm # open a new terminal and run

nvm install --lts   # To download and install the latest LTS release of NodeJS

nvm alias default lts/*  #Set the installed NodeJS as the default environment


# verify

node -v


  1. [node, version, nvm] How to change the node version using nvm (ref)


nvm use 8.9.4


  1. [raspbeberry, clone, dev] How to clone your development environment (ref)

  2. [node, process, argv] How to get the argument (view)

  3. [node, process, exit] How to exit the node.js -- process.exit() (ref)

  4. [node, process, child] Child Process in Node (slide)

  5. [node, server, helloworld] How to write a hello world (view)

3.5. Docker

  1. [node, docker] How to run your node app in docker (view) (ref)

  2. [node, docker, common] An example for the docker run script (view)

3.6. Watch

  1. [node, reload] How to autoreload a nodejs server code 


Step 1: Install monitor

sudo npm install nodemon -g


Step 2: run the monitor

nodemon your_server_site.js


Step 3:(optional) Increase the watcher (ref)

echo fs.inotify.max_user_watches=582222 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p




3.7. Buffer

  1. [Buffer] definition


Buffer objects are used to represent a fixed-length sequence of bytes.


  1. [Buffer, declare] How to use Buffer in Typescript (ref)


Getting error TS2304: Cannot find name 'Buffer'

==>

yarn add @types/node


  1. [Buffer, create] How to create a Buffer


let byteBuffer:Buffer = Buffer.alloc(0);

  1. [Buffer, merge] How to merge two Buffers


data = Buffer.concat([data, chunk]);

  1. [Buffer, Buffer2string] How to conver Buffer to Hex string (view)


let buf:Buffer = Buffer.from([1, 2, 3]);

console.log('Data in Hex = ', buf.toString('hex')

  1. [Buffer, buffer2typedarray] How to create a Int16Array from Buffer (ref)


let int16Arr:Int16Array = new Int16Array(bufData);

  1. [Buffer, buffer2U32int] How to conver a 4 bytes Buffer to an unsigned 32 integer value (view)


let bufData:Buffer = Buffer.from([0, 0x71, 0x02, 0x00]);

let uint32Data:Uint32Array = new Uint32Array(bufData);

let uint32value:number =(uint32Data[0]& 0xff) | (uint32Data[1] & 0xff) << 8 | (uint32Data[2]&0xff) << 16 | (uint32Data[3]*0xff) << 24;


console.log("uint32value = ", uint32value); // should be 160000



3.8. Process

  1. [process, doc] A tutorial on command line (view)

  2. [process, pwd] How to launch a system command, pwd (view)

  3. [process, pipe] How to use pipe in spawn (view)

  4. [process, execFile] How to launch an execution file (view)

  5. [process, execFile] How to automatic insert command to interactive program -- use execFile (view)

  6. [command line, exec] How to launch an execution file from shell (view) (replace by exec command)

  7. [command line, auto inerst] How to auto insert command to interactive program (view)

  8.  [process, benchmark, console] How to use console.time to benchmark your code (ref)


console.time("Algorithm ABC benchmarking");

....

console.end("Algorithm ABC benchmarking")


  1. [process, hrtime] How to evaluate the execution time in high-resolution realtime, nanoseconds (view)

    1. NodeJS

const start:bigint = process.hrtime.bigint();

const end:bigint = process.hrtime.bigint();

console.log(`Benchmark took ${end - start} nanoseconds.`);

  1. Browser

this.__start = hrtime();

...

const diff:[number, number] = hrtime(this.__start);

this.setState({diff:diff});


// render

<h2>Benchmark took {this.state.diff[0] * 1e9 + this.state.diff[1]} nanoseconds</h2>




  1.  [process, node] How to determine the script run in browser or node? (ref)


if(typeof process === 'object'){

    // in node

}

  1. [process, python] How to communicate between Python and NodeJs (ref)


4. Libraries

4.1. JSON

  1. [schema validator, ajv, official] How to use ajv to validate the request JSON (view) (ref)

  2. [schema validator, is-my-json-valid] How to use /is-my-json-valid to validate the request JSON (view)


4.2. CRUD

  1. [crud, canjs] The canjs official site (ref)


4.3. Asynchronous

4.3.1. Promise

  1. [async, promise] How to use promise (view)

  2. [async, await, promise, then] 當你需要大時間處理的時候,就是使用 promise 搭配 async 與 wait 的時候. 非同步指令的執行順序性如何確保等重要問題,可以使用 一連串 await 或者是使用 一連串的 then 來達成。(promises) (view)

  3. [async] How to use async new command (view)

  4. [async, await] The async function usage examples, typescript (view)***


  5. [async, listener] How to async await function in a listener?


Key

Add the async keyword to the anonymous arrow inline function.

     

Example 

class MyClass{

    ....

    initButton() {

        let button = document.createElement('button');

        button.innerHTML = 'LongRun';

        button.addEventListener('Play', async () => {

             this.buffer = await this.longRunTask();  // <-- wait task finished

             this.playSound(this.buffer);

       });

    }

}



4.3.2. Stream, RxJS

  1. [rxjs, official] The official site (ref)

  2. [rxjs, java] RxJava (ref)

  3. [rxjs, tutorial] The introduction to Reactive Programming you've been missing (ref)** 


  • Almost everything can be a stream (ref)

  • Observed: The stream is the subject (or "observable") being observed.

  • Map: The map(f) function which takes each value of stream A, applies f() on it, and produces a value on stream B.

  • Scan: The scan(g) function aggregates all previous values on the stream, producing value x = g(accumulated, current), where g was simply the add function in this example.

  • Throttle:

  1. [rxjs, why] 


  • Reactive Programming raises the level of abstraction of your code so you can focus on the interdependence of events that define the business logic, rather than having to constantly fiddle with a large amount of implementation details. (ref)

  1. [rxjs, promise] A Promise is simply an Observable with one single emitted value. Rx streams go beyond promises by allowing many returned values. (ref)

  2. [rxjs, promise] How to conver a Promise to an Observable (ref)

    1. A Promise is simply an Observable with one single emitted value

var stream = Rx.Observable.fromPromise(promise)

  1. [rxjs, observable, html] How to conver a button event to an observable object (ref)


// Example: Almost everything can be a stream

var refreshButton = document.querySelector('.refresh');

var refreshClickStream = Rx.Observable.fromEvent(refreshButton, 'click');


  1. [rxjs, stream, merged] How to merge two stream (ref)


var requestOnRefreshStream = refreshClickStream

  .map(function() {

    var randomOffset = Math.floor(Math.random()*500);

    return 'https://api.github.com/users?since=' + randomOffset;

  });

  

var startupRequestStream = Rx.Observable.just('https://api.github.com/users');


var requestStream = Rx.Observable.merge(

  requestOnRefreshStream, startupRequestStream

);


  1. [rxjs, hello] A rxjs helloword project (view)


import {Subject, Observable} from 'rxjs';


var observable = Observable.create(function (observer:any) {

    console.log('Hello');

    observer.next(42);

    observer.next(100); // "return" another value

    observer.next(200); // "return" yet another

  });


observable.subscribe(function (x:any) {

  console.log('subscribe: ' + x);

});


4.4. Data structure 

  1. Circular queue (ref)


4.5. Network

  1. [ip] How to get the ip address (view)

  2. [auto-reload] How to auto reload the page when the file was changed (view)

4.5.1. Mqtt

  1. [mqtt, concept] A quick note of MQTT (view), (good)

  2. [mqtt, protocol] MQTT V3.1 Protocol Specification (view)

    1. Publish packet (view)

  3. [mqtt, broker, survey] MQTT Brokers/Servers and Cloud Hosting Guide (ref)

  4. [mqtt, broker, public] HIVEMQ (view)

  5. [mqtt, tls/ssl] How to use tls/ssl for mqtt communication (view)

  6. [mqtt, authentication] How to use authentication in Mosca MQTT Broker (ref)

4.5.1.1. Plan Mqtt

  1. [mqtt, broker, aedes] How to create a mqtt broker using aedes (view

    1. [mqtt, broker, client, mosca] A simple broker and client (view), UNMAINTAIN

  2. [mqtt, publisher, mqtt] How to create a mqtt publisher using mqtt (view

  3. [mqtt, subscribe, mqtt] How to subscribe a topic using mqtt (view)

4.5.1.2. Mqtt over Websocket

  1.  [mqtt, broker, websocket, aedes] Setting up private MQTT broker under WebSocket using Aedes in Node.js with TypeScript (view)

  2. [mqtt, publisher, websocket, mqtt] How to publish a random sequence data to a MQTT broker over websocket (view)



4.5.2. Http server

  1. [http-server, load balanced] How to transfer a task to another child http server (view)

  2. [http, server] How to create a http server


var express = require('express');


const app = express();

const server = app.listen(3000, () => {

  const address = server.address();

  console.log(`http://localhost:${address.port}\n`);


  server.once('close', () => {

    connectionManagers.forEach(connectionManager => connectionManager.close());

  });

});



  1. [http, favicon.ico] How to work around the "GET http://localhost:8001/favicon.ico 404 (Not Found)" (ref)


// This solution was copied from Jason Ziegler    

if (req.url === '/favicon.ico') {

            res.writeHead(200, {'Content-Type': 'image/x-icon'} );

            res.end();

            console.log('favicon requested');

            return;

    } else {

        router.home(req, res);

        router.user(req, res);

  }

});



4.5.3. Net

  1. [doc] What are the different between socket.io and .net for the node modules (ref)


# The network stack 

socket.io

webSocket

TCP


  1. [socket, server] How to create a simple net server/client to communicate binary stream (view)


var net = require('net');

var client = new net.Socket();

client.connect(1337, '127.0.0.1', function() {

console.log('Connected');

client.write('Hello, server! Love, Client.'); // (ok) Test send text data

client.write(bufData);

});


  1. [socket, client] How to write a binary to server


// nodejs

var net = require('net');

var client = new net.Socket();

client.write(bufData);



4.5.4. Socket.io

  1. [socket.io] Socket.io 適合傳遞短訊息, 不適合傳遞大量 binary 資料

  2. [socket.io, cordova] Can we use socket.io in Codova? (ref)

  3. [socket.io, text] How to communication p2p with text (ref) (view)


// Server

...

io.on('connection', (socket) => {

  console.log('a user connected');

  socket.on('chat message', (msg) => {

    console.log('message: ' + msg);

  });

  

  socket.on('disconnect', () => {

    console.log('user disconnected');

  });

});



// Client

 var socket = io();

 socket.emit('chat message', 'hello');



  1. [socket.io, stream, text] How to streaming text from Android to a nodejs socket.io server (view)

  2. [socket.io, binary] How to send binary data to server (view)


// Server

...

io.on('connection', (socket) => {

  console.log('a user connected');

  socket.on('chat message', (msg) => {

    console.log('message: ' + msg);

  });

  

  socket.on('disconnect', () => {

    console.log('user disconnected');

  });

});



// Client

 var socket = io();

 <script>

        var socket = io();

        var buf = new Uint8Array([1, 129, -1, 0x0a]);

        socket.emit('chat message', 'hello');

        var i;

        for (i = 0; i < buf.length; i++) {

          socket.emit('chat message', buf[i]);

        }

    </script>




  1. [socket.io, streaming, audio] How to streaming the audio file to browser (view)


// Server

// server.js

var io = require('socket.io'); // 加入 Socket.IO

fs.readFile(strAudiofile, function(error, byteBuffer){

    console.log(byteBuffer.length);

    socket.emit('data_wave', byteBuffer);

  });



// View

// render.html

<script src="/socket.io/socket.io.js"></script>

<button onclick="myFunction()">play stream</button>


<script>

    function myFunction() {

       ....

       var socket = io.connect();

            

                socket.on('data_wave', function(arrayBuffer){

                    console.log('data_wave');

                    var blob = new Blob([arrayBuffer], { 'type' : 'audio/wav' });

                    var audio = document.createElement('audio');

                    audio.src = window.URL.createObjectURL(blob);

                    audio.play();

    }

}

  1. [socket.io, base64] How to received a base64 binary data (view)


// Android site

strData_base64 = Base64.encodeToString(byteBuffer, Base64.DEFAULT);

mSocket.emit("binary_data", strData_base64);


// node site

socket.on('binary_data', (strMessage_Base64) => {

    console.log('strMessage_Base64: ', strMessage_Base64);

    let lstByteChunk = new Buffer (strMessage_Base64, 'base64');


    console.log('lstByteChunk: ', lstByteChunk);

    var i = 0;

    for(i = 0; i < lstByteChunk.length; i++){

      console.log(i, 'lstByteChunk[i] = ', lstByteChunk[i]);

    }

    q.push(lstByteChunk);

  });





4.6. HTTP Client

  1. [post, contenttype] What is ContentType (ref)

    1. The HTTP POST method sends data to the server. The type of the body of the request is indicated by the Content-Type header

  2. [post, form] Which ContentType can be used to sent a post request?


  1. application/x-www-form-urlencoded: not suitable to use with binary data. Example: key1=value1 & key2=value2 & key3=value3

  2. multipart/form-data:  each value is sent as a block of data ("body part"), with a user agent-defined delimiter ("boundary") separating each part.


  1. [post, sendfile, binary, body, base64, needle] How to send a post request to remote site via body base64 string

    1. Content-Type: application/json; charset=utf-8 

    const content = fs.readFileSync("./samples/test.pdf");

    const base64Data = Buffer.from(content).toString('base64');


    // process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'

    const postData = {

      Key: 'test_v1',

      No: 'test_v1',

      FileType: 'pdf',

      Base64: base64Data,

    }


    /*

    Server Site: 

    ctx.header.content-type: "application/json; charset=utf-8"

    ctx.request: ...

    ctx.request.body: {your data} (<--)

    ctx.req: ...

    ctx.req.body: undefined

    ctx.req.files: undefined  (<--) 

    ctx.body: undefined

    */

    needle

        post( 'http://localhost:13088/upload', postData, { rejectUnauthorized: false, json: true })


  1. Content-Type: application/x-www-form-urlencode 

    const content = fs.readFileSync("./samples/test.pdf");

    const base64Data = Buffer.from(content).toString('base64');


    // process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0'

    const postData = {

      Key: 'test_v1',

      No: 'test_v1',

      FileType: 'pdf',

      Base64: base64Data,

    }



    /*

    Server Site: 

    ctx.header.content-type: "application/x-www-form-urlencoded"

    ctx.request: object

    ctx.request.body: {Your data} (<--)

    ctx.req: IncomingMessage...

    ctx.req.body: undefined  (<--) 

    ctx.req.files: undefined  (<--) 

    ctx.body: undefined

    */


    needle

        post('http://localhost:13088/upload', postData)



  1. [post, sendfile, binary, attach, needle] How to send a post request to remote site via body base64 string

    1. Content-Type: multipart/form-data

var data = {

        Key: 'test-v1',

        No: 'carPlateNo-v1',

        file: fs.readFileSync('./samples/test.pdf'),

      };

      

      /*

    Server Site: 

    ctx.header.content-type: "multipart/form-data; boundary=--------------------------414518500358049776530516"

    ctx.request: Object {app: Application ...}

    ctx.request.body: Object {}

    ctx.req: IncomingMessage 

    ctx.req.body: {Your Data}  (<--)

    ctx.req.files: Array(1) [Object]  (<--) 

    ctx.req.files.0: mimetype: "application/octet-stream"

    ctx.body: undefined

    */

      needle

        .post('http://localhost:13088/upload', data, { multipart: true })

        .on('readable', function() { /* eat your chunks */ })

        .on('done', function(err, resp) {

          console.log('Ready-o!');

        })


  1. Content-Type



  1. [post, sendfile, txt, body, utf-8, needle]**** How to send a post request to remote site via body utf-8 string

    1. Content-Type: application/json

  const content = fs.readFileSync("./samples/test.json");

   const strData = Buffer.from(content).toString();


    var data = {

        Key: 'test-v1',

        No: 'carPlateNo-v1',

        file: strData,

      };


     /*

    Server Site: 

    ctx.header.content-type: "application/json"

    ctx.request: Object (app: Application, ...)

    ctx.request.body: your data}  (<--)

    ctx.req: IncomingMessage...

    ctx.req.body: undefined   (<--) 

    ctx.req.files: undefined  (<--) 

    

    ctx.body: undefined

    */

  

      needle

        .post('http://localhost:13088/upload', data, {json:true })

        .on('readable', function() {  })

        .on('done', function(err, resp) {

          console.log('Ready-o!');

        })

  



  1. [post, needle] How to send a post to remote site? (ref)

    1. Client

var data = {

  file: '/home/johnlennon/walrus.png',

  content_type: 'image/png'

};


// the callback is optional, and needle returns a `readableStream` object

// that triggers a 'done' event when the request/response process is complete.

needle

  .post('https://my.server.com/foo', data, { multipart: true })

  .on('readable', function() { /* eat your chunks */ })

  .on('done', function(err, resp) {

    console.log('Ready-o!');

  })


  1. Server

# server side

1. ctx.req.body.file = '/home/johnlennon/walrus.png'

2. cts.req.body.content_type:'image/png'

3. cts.req.files = []   // <<--- no files


  1. [post, sendfile, form-data] How to send a post request to remote site with two files using WebAPI FormData (ref)


// node

//   yarn add form-data

//   var FormData = require('form-data');

//   var fs = require('fs');


var form = new FormData();

form.append('key1', 'mykey1');

form.append('my_file', fs.createReadStream(__dirname + "/file1.jpg"));

form.append('my_file', fs.createReadStream(__dirname + "/file2.jpg"))


form.submit(frontend + "/post_img", function(err, result) {

    // res – response object (http.IncomingMessage)  //

    console.log(result);

});


  1. Asdf

  1. [post, postman] How to send files to the remote site using postman based on HTTP post request

    1. Body: form-data: file [browsing] 

    2. (edit)


4.7. Html Parser

4.7.1. JQuery

  1. [jquery, page] JQuery (ref)

  2. [jquery, table, tabulator] [table, load] How to load data to tabulator via json format ? (ref), (ref2)

  3. [table, tabulator] Beautiful Table (ref)

  4. [table, load] How to load data to tabulator via json format ? (ref), (ref2)

  5. [table, edit] How to create a editable table (ref)

4.7.2. htmlparser2

  1. [htmlparser2, html, streaming, typescript] How to streamly parse an url response using htmlparser2 (view)


Key

  1. How to get the html readable streaming

    1. https.get(strUrl, resCallback)


  1. [htmlparser2, parse, 玉山, typescript] 如何使用串流模式解析 HTML Table, 以解析玉山國家公園違規名單為例 (view)

4.7.3. cheerio

  1. [cheerio, parsing, table, typescript] How to parse html table (view)


4.8. Bluetooth

  1. [ble, service] The bluetooth standard service (ref)

  2. [bluetooth, spp] How to use Bluetooth serial port communication for Node.js (view)

  3. [ble, scan] How to scan a BLE device (view)

  4. [ble, connect] How to connect to a device and discovery all services and characters (view)

  5. [ble, characteristic] How to compare the characteristic uuid? (view)


Key

if (characteristic.uuid === '00000004000010008000008025000000'){

       console.log('found 00000004000010008000008025000000')

}


  1. [ble, characteristic] How to get the characteristics for a BLE device (view)


Key:

使用 async function 來 await BLE peripheral.discoverAllServicesAndCharacteristics resolve 資料.




5. Backend

5.1. Security

  1. [cern] 免費憑證: Let’s Encrypt、SSL4Free (ref)

  2. [cern, cer] How to conver a cer file from der format to PEM format


openssl x509 -inform der -in 38C235C9F680AF165D61DBAAB3282526.cer -out 38C235C9F680AF165D61DBAAB3282526.pem


#x509

PEM formated file is a text file begin with "BEGIN"


  1. [cors] The Complete Guide to CORS (In)Security (ref)

5.2. Web Server

  1. [webserver, ngnix] Introduction to the Ngnix (view)

  2. [ngix, rewrite] How to rewrite http request to https (ref) (ref2)


step 1: ssh my-host-ip

step 2: vi /etc/nginx/sites-enabled/default

step 3: add the following line

rewrite ^(.*) https://$host$1 permanent;


sudo nginx -t  # test config                

sudo systemctl restart nginx # restart 


  1. [ngix, error] Error: ENOENT: no such file or directory, open tmp/xxxxx. ... (ref)


It complains tmp does not exists. Create tmp foder.




5.3. Database


5.3.1. Comm

  1. [database] Key-value based Database vs. Document-based Database (view)



5.3.2. Mongodb

  1. [mongodb, mysql] MongoDB vs. MySQL (ref)


5.4. Flask

  1. Why server code using Python? Because it is related to IoT, Image processing, and Mathematics application (ref)

  2. [flask, jquery] How to combine jquery in flask (ref)

  3. [flask, react] React and Flask Full Stack Web app: Component-oriented and Data-driven (ref)

  4. [flask, react] Build a Simple CRUD (Create, read, update and delete) App with Python, Flask, and React (ref)



5.5. KOA/Express

  1. [sheet] The Express sheet (view)

  2. [express, debug] 

    1. [cors] 每次一定要重新開啟瀏覽器, 再測試. 不然 cors 每次的結果不一定會反應.

  3. [express, listen] How to listen two different ports (ref)


Code

const express = require('express');

var app = express();

var admin_app = express();

const server1 = app.listen(3001, () => {

  const address = server1.address();

  console.log(`http://localhost:${address.port}\n`);

});


const server2 = admin_app.listen(3001, () => {

  const address = server2.address();

  console.log(`http://localhost:${address.port}\n`);

});


Output

http://localhost:3000

http://localhost:3001



  1. [express, deploy] How to deploy the React App with Express (view)

  2. [express, request] How to show the request headers infomation for debugging (view)


console.log(JSON.stringify(req.headers));

  1. [express, content-type] How to print the request's content-type (ref)


let myContentType = req.get('content-type');

console.log('myContentType = ' + myContentType);


  1. [express, body] How to retrieve/parse the data from req.body (view)


# use body parser

## content-type: application/json

router.post('/info', bodyParser.json(), function(req:Request, res:Response, next){

   ...

}


  1. [express, ejs] How to use view engine, ejs (ref)

  2. [express, author] How to handle user authentication using Auth0 in express (ref)

  3. [koa, simple] How to debug a client site typescript code in vscode with koa backend (view)


# Step 1: Open the project folder in VSCode

$code ./

vscode::[open] -> src/app/app.ts

vscode:: setup break points


# Step 2: start the app server

$cd ./src

$tsc     #compile all ts code to js

$node http-server.js


# Step 3: start to debug

vscode::[Debug] -> Launch Chrome against localhost


  1. [koa, static] How to mount a static middleware to express service?


app.use('/static', express.static(path.join(__dirname, 'public')))


把 public 目錄, 掛在 /static 下.


Q: app.use('/static', express.static('public'))?

==> create a virtual path prefix /stastic for files that are served by the express.static function. 

==> http://localhost:3000/static/images/kitten.jpg


  1. [koa, get] How to create a GET router handle middle function on the '/user/:id'?


app.get('/user/:id', function (req, res, next) {

  res.send('USER')

})


  1. [koa, use] How to get exectuion of the middle function for any type of HTTP request on the '/user/:id'?


app.use('/user/:id', function (req, res, next) {

  console.log('Request Type:', req.method)

  next()        // <---- pass control to the next middleware function

})



  1. [koa, request] ctx.request vs. ctx.req (ref)


// ctx.request: A koa request object

// ctx.req: Node's request object.


app.use(async ctx => {

  ctx; // is the Context

  ctx.request; // is a Koa Request

  ctx.response; // is a Koa Response

});


  1. [koa, config] A config example (view)



5.6. Admin (User) Management

  1. [admin, example] CRM Starter Admin App built with react and Ant Design (ref)

  2. A Free React Admin Template With Material Design And Tailwind CSS (ref)

5.7. Auth: for User Signup & User Login

  1. [auth, main] Auth: for User Login FAQ (view)



6. Hybrid App

  1. [main, hybrid] The Hybrid App Main Page (view)

7. Others

  1. Intel nw.js, application 利器, http://docs.nwjs.io/en/latest/