2022年6月16日 星期四

[mqtt, tls/ssl, websocket, mosquitto] How to setup a mqtt communication over websocket with TLS/SSL using mosquitto

How to setup a mqtt communication over websocket with TLS/SSL using mosquitto

井民全, Jing, mqjing@gmail.com

This document shows you how to setup a mqtt communication over websocket with TLS/SSL using mosquitto. In order to test the mqtt with secure https TLS/SSL over websocket.

The broker certification, server.crt, I use self-singed. The CA that i created for my self. The webserver, I use the KOA that using the same certification file server.crt for https.


Here are the following procedures:

  1. Create a CA key-pair for self-signed testing

    1. Output: 3 files: ca.crt and self-signed server.key and server.crt

  2. Install a mosquitto server for the mqtt broker

    1. Configuration: ca.crt, server.key and server.crt

    2. Output: a mqtt broker over websocket with TLS/SSL

  3. Install a KOA javascript web server for the javascript container that provide secure https TLS/SSL with the self-signed certification server.key and the server.crt

    1. Import the  ca.crt to the Chrome browser for trusting the Customized CA.

  4. Testing procedures for ensure: 

    1. paho utilities:  mosquitto_pub and mosquitto_sub are used to test the broker

    2. Python code: are used to demo and test the mqtt communication with the broker.

    3. Html + Javascript code: are used to simulate a mqtt client that running in a browser to communicate with the broker over websocket with secure https TLS/SSL. Here, I use Eclipse Paho Javascript Client library.


Table of Contents

1. Quick 2

1.1. Step 1: Install/Setup the mosquitto server 2

1.2. Step 2: Setup the TLS security 3

1.3. Step 3: Control the broker services 3

1.4. Step 4: Test 3

1.4.1. Command Line Test 4

1.4.2. Python Test 4

1.5. Trobule shotting 5

2. Enable TLS over websocket 5

3. Detail for Setup the TLS security 6

3.1. Setup a self-signed CA key-pair and the certification 6

3.2. Create key-pair for broker server 6

3.3. Create server certification using the "ca.key" 7

3.4. Edit the configure file 8

4. Start the Broker 8

5. Testing 9

5.1. Command Line 9

5.1.1. MQTT Client (Normal) 9

5.1.2. MQTT Client (with TLS) 9

5.2. Python Client (TLS) 9

5.2.1. Requirement 9

5.2.2. Publish 9

5.2.3. Subcriber 10

5.3. Python Client (TLS + WebSockets) 12

5.3.1. Requirement 12

5.3.2. Publish 12

5.3.3. Subcriber 13

5.4. Javascript Client (TLS + WebSocket) 15

5.4.1. The file structure 15

5.4.2. (Only for self-signed case) Add the customized CA cert to the Chrome browser 15

5.4.3. Setup a web server with https enabled 17

5.4.4. Html 18

5.4.5. Javascript 18

6. Trouble-shooting 23

6.1. Error messages 23

6.1.1. Message: error 18 at 0 depth lookup:self signed certificate (ref) 23

6.1.2. Message: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain 24

6.1.3. Message: mqttws31.min.js:36 WebSocket connection to 'wss://xxx.xxx.xxx.xxx:8081/mqtt' failed: Error in connection establishment: net::ERR_CERT_AUTHORITY_INVALID 24

7. References 24


1. Quick

1.1. Step 1: Install/Setup the mosquitto server

# install

sudo apt-get install mosquitto mosquitto-clients

cd /etc/mosquitto/conf.d/


# get the config

sudo wget https://github.com/eclipse/mosquitto/blob/master/mosquitto.conf /etc/mosquitto/conf.d/ mosquitto.conf


sudo vi mosquitto.conf


File: /etc/mosquitto/conf.d/mosquitto.conf

# for testing

allow_anonymous true

listener 1883


# for websockets

listener 9001

protocol websockets


# for TLS

listener 8883

cafile /etc/mosquitto/ca_certificates/ca.crt

keyfile /etc/mosquitto/certs/server.key

certfile /etc/mosquitto/certs/server.crt




1.2. Step 2: Setup the TLS security

mkdir cert2; cd cert2


# The operations for Self CA

openssl genrsa -des3 -out ca.key 2048

openssl req -new -x509 -days 1826 -key ca.key -out ca.crt # for self-signed, leave all empty



# for broker server

openssl genrsa -out server.key 2048

openssl req -new -key server.key  -out server.csr    # using server.key to encrypt server.csr

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360 # Common Name: using the broker doman name or ip (Don't let the value same as the ca)


# verification the broker server crt (very important)

openssl verify -CAfile ca.crt server.crt


# deploy the certifications

sudo cp ca.crt  /etc/mosquitto/ca_certificates

sudo cp server.key server.crt /etc/mosquitto/certs


1.3. Step 3: Control the broker services

# control the broker service

sudo service mosquitto stop

sudo service mosquitto start

sudo service mosquitto restart


1.4. Step 4: Test

Download the Windows binary 

https://mosquitto.org/download/


# On the other machines

1.4.1. Command Line Test

# (1) MQTT Client (Normal)

# publish (172.20.10.3: is the broker ip)

"C:\Program Files\mosquitto\mosquitto_pub" -h 172.20.10.3 -m "test message" -t vital/ppg


# subscribe

"C:\Program Files\mosquitto\mosquitto_sub" -h 172.20.10.3 -t vital/ppg -C 3


# (2) MQTT Client (with TLS)

# publish to broker

"C:\Program Files\mosquitto\mosquitto_pub" -h 172.20.10.3 -t vital/ppg -m "test message" --cafile .\ca.crt -p 8883



# subscribe

"C:\Program Files\mosquitto\mosquitto_sub" -h 172.20.10.3 -t vital/ppg --cafile .\ca.crt -p 8883  -C 3 


1.4.2. Python Test

pip install paho-mqtt


python subscribe_tls.py  (source)

python publish_tls.py   (source)




1.5. Trobule shotting

# log

sudo cat /var/log/mosquitto/mosquitto.log


# verification the broker server crt (very important)

openssl verify -CAfile ca.crt server.crt


2. Enable TLS over websocket

Step 1: vi the mosquitto.conf

Enable the websockets

File: /etc/mosquitto/conf.d/mosquitto.conf

listener 8081

protocol websockets

cafile /etc/mosquitto/ca_certificates/ca.crt

keyfile /etc/mosquitto/certs/server.key

certfile /etc/mosquitto/certs/server.crt

Step 2: restart the broker

sudo service mosquitto restart


Step 3: Testing

Subscriber (source)

python 03.03-test-subscribe-tls-websocket.py


Publish (source)

python 03.04-test-publish-tls-websocket.py


3. Detail for Setup the TLS security


3.1. Setup a self-signed CA key-pair and the certification

Step 1: Create a key pair for the self-signed CA

Command

openssl genrsa -des3 -out ca.key 2048

Step 2: Create a the ca.cert using the ca.key

Command

openssl req -new -x509 -days 1826 -key ca.key -out ca.crt

Common Name: CA-DOMAIN NAME (if you want to test self-signed, leave empty)


3.2. Create key-pair for broker server

Step 1: Create the server.key

openssl genrsa -out server.key 2048

Step 2: Create the server.csr

openssl req -new -out server.csr -key server.key

Common Name: using your broker server's domain name


3.3. Create server certification using the "ca.key"

Step 1: Create server.crt using self-signed CA using the ca.key 

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360


Verify the server.crt

Command

openssl verify -CAfile ca.crt server.crt

Result


Step 2: Copy the certifications and the server key


// non-Docker version

sudo cp ca.crt  /etc/mosquitto/ca_certificates

sudo  cp server.key server.crt /etc/mosquitto/certs



// Docker version

cp ca.crt /home/jing/work/mosquitto/ca_certificates

cp server.key server.crt /home/jing/work/mosquitto/certs


3.4. Edit the configure file

vi mosquitto.conf 

Listener 8883 

cafile /etc/mosquitto/ca_certificates/ca.crt

keyfile /etc/mosquitto/certs/server.key

certfile /etc/mosquitto/certs/server.crt


4. Start the Broker

sudo service mosquitto restart

// Debug

sudo cat /var/log/mosquitto/mosquitto.log


5. Testing

5.1. Command Line

5.1.1. MQTT Client (Normal)

# publish (172.20.10.3: is the broker ip)

"C:\Program Files\mosquitto\mosquitto_pub" -h 172.20.10.3 -m "test message" -t vital/ppg


# subscribe

"C:\Program Files\mosquitto\mosquitto_sub" -h 172.20.10.3 -t vital/ppg -C 3



5.1.2. MQTT Client (with TLS)

# publish to broker

"C:\Program Files\mosquitto\mosquitto_pub" -h 172.20.10.3 -t vital/ppg -m "test message" --cafile .\ca.crt -p 8883



# subscribe

"C:\Program Files\mosquitto\mosquitto_sub" -h 172.20.10.3 -t vital/ppg --cafile .\ca.crt -p 8883  -C 3 




5.2. Python Client (TLS)

5.2.1. Requirement

pip install paho-mqtt


5.2.2. Publish

File: publish_tls.py

import paho.mqtt.client as mqtt

import ssl


strbrokerIP = "172.20.10.3"

strTopic = "vital/ppg"


def on_connect(client, userdata, flags, rc):

    print("Connected with result code "+str(rc))

    client.subscribe(strTopic)


def on_message(client, userdata, msg):

    print(msg.topic+" "+ msg.payload.decode('utf-8'))


def publish(numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼


    client.connect(strbrokerIP, numPort, 60)

    client.publish(strTopic, "Hello World")


def publish_tls(strCACRT, numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.publish(strTopic, "Hello World")


def main():

    # subscribe(1883);

    # publish(1883);


    strCA_crt = "./ca.crt"

    publish_tls(strCA_crt, 8883)


main();



5.2.3. Subcriber

File: subscriber_tls.py


import paho.mqtt.client as mqtt

import ssl


strbrokerIP = "172.20.10.3"

strTopic = "vital/ppg"


def on_connect(client, userdata, flags, rc):

    print("Connected with result code "+str(rc))

    client.subscribe(strTopic)


def on_message(client, userdata, msg):

    print(msg.topic+" "+ msg.payload.decode('utf-8'))


def subscribe(numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼


    client.connect(strbrokerIP, numPort, 60)

    client.loop_forever()


def subscribe_tls(strCACRT, numPort, ):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    # client.tls_set(strCACRT)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.loop_forever()


def main():

    # subscribe(1883);


    strCA_crt = "./ca.crt"

    subscribe_tls(strCA_crt, 8883)

main();



Result

python subscribe_tls.py

python publish_tls.py


5.3. Python Client (TLS + WebSockets)

5.3.1. Requirement

pip install paho-mqtt

5.3.2. Publish


import paho.mqtt.client as mqtt

import ssl


strbrokerIP = "172.20.10.3"

strTopic = "vital/ppg"


def on_connect(client, userdata, flags, rc):

    print("Connected with result code "+str(rc))

    client.subscribe(strTopic)


def on_message(client, userdata, msg):

    print(msg.topic+" "+ msg.payload.decode('utf-8'))


def publish(numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼


    client.connect(strbrokerIP, numPort, 60)

    client.publish(strTopic, "Hello World")


def publish_tls(strCACRT, numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.publish(strTopic, "Hello World")


def publish_tls_websocket(strCACRT, numPort):

    client = mqtt.Client(transport='websockets')

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.publish(strTopic, "Hello World")


def main():

    # subscribe(1883);

    # publish(1883);


    # strCA_crt = "./ca.crt"

    # publish_tls(strCA_crt, 8883)


    

    strCA_crt = "./ca.crt"

    publish_tls_websocket(strCA_crt, 1881)



main();



5.3.3. Subcriber

import paho.mqtt.client as mqtt

import ssl


strbrokerIP = "172.20.10.3"

strTopic = "vital/ppg"


def on_connect(client, userdata, flags, rc):

    print("Connected with result code "+str(rc))

    client.subscribe(strTopic)


def on_message(client, userdata, msg):

    print(msg.topic+" "+ msg.payload.decode('utf-8'))


def subscribe(numPort):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼


    client.connect(strbrokerIP, numPort, 60)

    client.loop_forever()


def subscribe_tls(strCACRT, numPort, ):

    client = mqtt.Client()

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    # client.tls_set(strCACRT)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.loop_forever()


def subscribe_tls_websocket(strCACRT, numPort, ):

    client = mqtt.Client(transport='websockets')

    client.on_connect = on_connect

    client.on_message = on_message

    # client.username_pw_set("try","xxxx") # 設定登入帳號密碼

    client.tls_set(strCACRT, tls_version=ssl.PROTOCOL_TLSv1_2)

    # client.tls_set(strCACRT)

    client.tls_insecure_set(True)

    


    client.connect(strbrokerIP, numPort, 60)

    client.loop_forever()


def main():

    # subscribe(1883);


    # strCA_crt = "./ca.crt"

    # subscribe_tls(strCA_crt, 8883)


    strCA_crt = "./ca.crt"

    subscribe_tls_websocket(strCA_crt, 8081)

main();



5.4. Javascript Client (TLS + WebSocket)

5.4.1. The file structure

├── public

│   ├── index.html

│   └── index.js

├── server.crt

├── server.key

└── webserver.js

   



5.4.2. (Only for self-signed case) Add the customized CA cert to the Chrome browser

Note: If you miss this step, you will got the the error of WebSocket connection to 'wss://xxx.xxx.xxx.xxx:8081/mqtt' failed.

Step 1: [Setting] - [Privacy and security] -> [Security]

Step 2: Check [Manage certificates]

Step 3: import your customized CA certification file: the ca.crt

For Windows version

[Personal] -> [Import...]

For Linux version

[Authorities] -> [Import...]

5.4.3. Setup a web server with https enabled

Step 1: Install KOA

# install n

curl -L https://git.io/n-install | bash


// or automatic install version (without ask)

curl -L https://git.io/n-install | bash -s -- -y  


// usage 

n 12


#uninstall n

# $n-uninstall -y



npm i koa


Step 2: Create start js code

Note: the key and certification using the same as broker. (if you use self-signed). If you using different server.cer, you will got the error of WebSocket connection to 'wss://xxx.xxx.xxx.xxx:8081/mqtt' failed.

File: ./webserver.js

const http = require("http");

const https = require("https");

const fs = require("fs");


const serve = require('koa-static')

const Koa = require('koa');

const path = require('path')

const app = new Koa();


console.log('__dirname = ' + __dirname);

app.use(serve(path.join(__dirname, '/public')))


http.createServer(app.callback()).listen(3000);

console.log('http, listening on port 3000');


const options = {

    key: fs.readFileSync("./server.key", "utf8"),

    cert: fs.readFileSync("./server.cet", "utf8")

};

https.createServer(options, app.callback()).listen(3001);

console.log('https, listening on port 3001');


Step 3: start the web server

node webserver.js

5.4.4. Html

File:./public/index.html

<!doctype html>

<html lang="en">


<head>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.min.js" type="text/javascript"></script>

</head>


<body>

    <h1>Test</h1>

    <script src="index.js"></script>

</body>

</html>


5.4.5. Javascript

File: ./public/index.js

// API Doc: https://www.eclipse.org/paho/files/jsdoc/Paho.MQTT.Client.html


'use strict';


let strBrokerIP = "172.20.10.3"; // "192.168.1.108"; // "172.20.10.3";

let eleInput = null;

let strTopic = "vital/ppg";

let client = null;

let message = null;

let body = document.getElementsByTagName("body")[0];

let eleStatus = null;



function createHtmlElement(strStyle, strTitle, funClickCallback){

  let ele = document.createElement(strStyle);

  if(strTitle != null)

    ele.innerHTML = strTitle;

  if (funClickCallback != null)

    ele.addEventListener('click', funClickCallback);

  return ele;

}


function createHtmlLabel(strTitle, body){

  let ele = createHtmlElement('label', strTitle, null);

  if(body == null){

    console.log('Error: createHtmlPTag::body == null')

    return null;

  }

  body.appendChild(ele);

  return ele;

}


function createHtmlInput(strTitle, body){

  let ele = createHtmlElement('input', null, null);

  ele.type = "text";

  ele.defaultValue = strTitle;

  if(body == null){

    console.log('Error: createHtmlBr::body == null')

    return null;

  }


  body.appendChild(ele);

  return ele;

 

}


function createHtmlBr(body){

  let ele = createHtmlElement('br', null, null);

  if(body == null){

    console.log('Error: createHtmlBr::body == null')

    return null;

  }


  body.appendChild(ele);

  return ele;

}


function createHtmlButton(strtitle, funClickCallback, body){

  let ele = createHtmlElement('button', strtitle, funClickCallback);

  if(body == null){

    console.log('Error: createHtmlButton::body == null')

    return null;

  }


  body.appendChild(ele);

  return ele;

}


function createHtmlPTag(strTitle, body){

  let ele = createHtmlElement('p', strTitle, null);

  if(body == null){

    console.log('Error: createHtmlPTag::body == null')

    return null;

  }

  body.appendChild(ele);

  return ele;

}


function updatePTagTitle(eleStatus, strTitle){

  if(eleStatus == null){

    console.log('Error: eleStatus == null.');

    return;

  }

  eleStatus.innerText = strTitle;

  console.log('UpdatePTagTitle:: ' + strTitle)

}


function cbConnect(){

    console.log('cbConnect::start to connect and send message');


    // get the broker ip

    if (eleInput == null){

        document.write("cbConnect::Error: eleInput == null.")

        return;

    }

    strBrokerIP = eleInput.value;

    // document.write("cbConnect::strBrokerIP = " + strBrokerIP);

    updatePTagTitle(eleStatus, "cbConnect::strBrokerIP = " + strBrokerIP);


    let clientId = "clientId" + Math.floor(Math.random() * 10).toString();

    console.log('clientId = ' + clientId);

    client = new Paho.MQTT.Client(strBrokerIP, Number(8081), clientId);

    


    client.onConnectionLost = onMQTTConnectionLost;

    // client.connect({onSuccess:onMQTTConnect});

    var options = {

      useSSL: true,

      onSuccess: onMQTTConnect,

    };

    client.connect(options)

}


function cbSendRandomMsg() {

  if (client == null) {

    console.log('client == null. Not connected?')

    return;

  }


  let strMsg =  Math.floor(Math.random() * 10).toString();

  message = new Paho.MQTT.Message(strMsg);

  message.destinationName = strTopic;

  client.send(message);



  console.log("cbSendRandomMsg:" + strMsg)

  updatePTagTitle(eleStatus, "cbSendRandomMsg:" + strMsg);

}


function cbRegReceiver() {

  console.log('Register to received a messages');

  if (client == null) {

    console.log('client == null. Not connected?')

    return;

  }

  client.onMessageArrived = onMQTTMessageArrived;

}



function onMQTTConnect() {

  console.log("onConnect");

  updatePTagTitle(eleStatus, "onConnected");

  client.subscribe(strTopic);

}


function onMQTTConnectionLost(responseObject) {

  if (responseObject.errorCode !== 0) {

    console.log("onConnectionLost:"+responseObject.errorMessage);

    updatePTagTitle(eleStatus, "onConnectionLost:"+responseObject.errorMessage);

  }

}


function onMQTTMessageArrived(message) {

  console.log("onMessageArrived:"+message.payloadString);

  updatePTagTitle(eleStatus, "onMessageArrived:"+message.payloadString);

}


function uiMainBuild(){

  // Input: the broker ip

  createHtmlLabel("The Broker IP:", body);

  eleInput = createHtmlInput(strBrokerIP, body);

  createHtmlBr(body);


  // Build UI

  let btConnect = createHtmlButton("1. Connect", cbConnect, body);

  let BrTag = createHtmlBr(body);

  let btSend = createHtmlButton("2. Send Random number", cbSendRandomMsg, body);

  let btRegReive = createHtmlButton("(2). Reg receiver", cbRegReceiver, body);

  createHtmlBr(body);

  createHtmlBr(body);

  eleStatus = createHtmlPTag("Status", body);


  // Show status

  document.write(eleInput.value);

}


function main(){

  uiMainBuild();

}


main();



Result

6. Trouble-shooting

6.1. Error messages

6.1.1. Message: error 18 at 0 depth lookup:self signed certificate (ref)

Solution

Whatever method you use to generate the certificate and key files, the Common Name value used for the server and client certificates/keys must each differ from the Common Name value used for the CA certificate. Otherwise, the certificate and key files will not work for servers compiled using OpenSSL.


6.1.2. Message: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain

Solution

The root cause for this issue is because of the mismach between the server.crt and ca.crt. Try to makesure the matching these two certifications. You can use the following command for this.


openssl verify -CAfile ca.crt server.crt


6.1.3. Message: mqttws31.min.js:36 WebSocket connection to 'wss://xxx.xxx.xxx.xxx:8081/mqtt' failed: Error in connection establishment: net::ERR_CERT_AUTHORITY_INVALID

Solution

This is because you use self-signed certification that your browser don't recognized. Import the ca.crt to your browser. It should work.


7. References

  1. Mosquitto Official Site, https://mosquitto.org/

  2. Test your mqtt client, https://test.mosquitto.org/

  3. http://www.steves-internet-guide.com/mosquitto-tls/

  4. http://www.steves-internet-guide.com/mqtt-websockets/