User Tools

Site Tools


2017:09:04:tying-mqtt-websockets-and-nginx-together-with-docker

Tying MQTT, WebSockets, and Nginx together with Docker

Introduction

In a prior article, I showed you how to use Docker to run an always-up-to-date Nginx web server with encryption. In this guide, I'm going to expand upon that lesson by using Docker Compose to build a secure, real-time messaging, micro-service framework with the following attributes:

  • Nginx
    • Web server
  • Mosquitto message broker
    • WebSockets server
    • MQTT server
  • Additional client authentication backends

Prerequisites

Caveats

In the prior tutorial, I used `docker run` to launch the web server as a background service. Since I'm running multiple services together, I'll change that, and let `docker-compose` handle everything for me.


Setup source

mkdir -pv \
    ~/srv/mosquitto/ \
    ;

Mosquitto Dockerfile

Download Dockerfile to your ~/srv/mosquitto/ directory:

wget \
    -O ~/srv/mosquitto/Dockerfile \
    https://thad.getterman.org/_export/code/2017/09/04/tying-mqtt-websockets-and-nginx-together-with-docker?codeblock=2 \
    ;
Dockerfile
################################################################################

FROM		ubuntu:xenial
MAINTAINER	Louis T. Getterman IV <thad.getterman@gmail.com>

USER		root
WORKDIR		/root

################################################################################

# Arguments
ARG			mapURL
ARG			mongoURL

# Environment Variables
ENV			HOME /etc/mosquitto
ENV			DEBIAN_FRONTEND noninteractive

# Ports: MQTT broker insecure and secure
EXPOSE 1883 8883

# Ports: Websockets insecure and secure
EXPOSE 9001 9002

# Volumes: Mosquitto MQTT Broker
VOLUME		[ "/var/lib/mosquitto", "/var/log/mosquitto" ]

# Related Directories
RUN			mkdir -pv \
				/usr/share/doc/mosquitto/examples \
				/usr/lib/mosquitto-auth-plugin/ \
				;

################################################################################

# Prerequisite software packages
RUN			apt-get -y update \
			&& \
			apt-get -y install \
				\
				apt-file \
				atool \
				make \
				gcc \
				software-properties-common \
				wget \
				\
				libcurl4-openssl-dev \
				libhiredis-dev \
				libldap2-dev \
				libmysqlclient-dev \
				libpq-dev \
				libsqlite3-dev \
				libssl-dev \
			;

# Install software packages
# Note: you can use `apt-file list <package>` to view a package's file destinations.
RUN			apt-add-repository -y ppa:mosquitto-dev/mosquitto-ppa \
			&& \
			apt-get -y update \
			&& \
			apt-get -y install \
				\
				libmosquitto-dev \
				mosquitto \
				mosquitto-clients \
				mosquitto-dev \
			&& \
			apt-file update \
			;

# Download and prepare software for compiling
WORKDIR		/usr/local/src

# Download Mongo DB
RUN			wget -O mongo $mongoURL && \
				atool --extract mongo && \
				rm -rfv mongo && \
				mv -v mon* mongo \
				;

# Download Mosquitto Auth Plugin
RUN			wget -O map $mapURL && \
				atool --extract map && \
				rm -rfv map && \
				mv -v mos* map \
				;

# Compile Mongo DB
WORKDIR		/usr/local/src/mongo
RUN			./configure && make && make install

# Compile Mosquitto Auth Plugin
WORKDIR		/usr/local/src/map
RUN			cp -va config.mk.in config.mk \
			&& \
			sed -i "s/^BACKEND_CDB .*/BACKEND_CDB ?= yes/" config.mk && \
			sed -i "s/^BACKEND_MYSQL .*/BACKEND_MYSQL ?= yes/" config.mk && \
			sed -i "s/^BACKEND_SQLITE .*/BACKEND_SQLITE ?= yes/" config.mk && \
			sed -i "s/^BACKEND_REDIS .*/BACKEND_REDIS ?= yes/" config.mk && \
			sed -i "s/^BACKEND_POSTGRES .*/BACKEND_POSTGRES ?= yes/" config.mk && \
			sed -i "s/^BACKEND_LDAP .*/BACKEND_LDAP ?= yes/" config.mk && \
			sed -i "s/^BACKEND_HTTP .*/BACKEND_HTTP ?= yes/" config.mk && \
			sed -i "s/^BACKEND_JWT .*/BACKEND_JWT ?= yes/" config.mk && \
			sed -i "s/^BACKEND_MONGO .*/BACKEND_MONGO ?= yes/" config.mk && \
			sed -i "s/^BACKEND_FILES .*/BACKEND_FILES ?= yes/" config.mk && \
			\
			sed -i "s/^MOSQUITTO_SRC .*/MOSQUITTO_SRC = \/usr\/include/" config.mk && \
			sed -i "s/^OPENSSLDIR .*/OPENSSLDIR = \/usr\/include\/openssl/" config.mk \
			&& \
			make \
			&& \
			mv -v \
				/usr/local/src/map/auth*.so \
				/usr/lib/mosquitto-auth-plugin/auth-plugin.so \
			&& \
			mv -v \
				/usr/local/src/map/np \
				/usr/bin/ \
			;

# Clean-up
RUN			rm -rfv \
				/usr/local/src/* \
			;

# Link libraries
RUN			ldconfig

################################################################################

# Run
WORKDIR		$HOME
CMD			[ "/usr/sbin/mosquitto", "-c", "/etc/mosquitto/mosquitto.conf" ]

################################################################################

Docker Compose

Download docker-compose.yaml to your ~/srv/ directory:

wget \
    -O ~/srv/docker-compose.yaml \
    https://thad.getterman.org/_export/code/2017/09/04/tying-mqtt-websockets-and-nginx-together-with-docker?codeblock=4 \
    ;
docker-compose.yaml
---

version: '2'

services:

################################################################################

    web:

        container_name: nginx
        image: nginx:latest
        restart: always

        ports:
            - 80:80      # HTTP
            - 443:443    # HTTPS

        volumes:
            - ~/srv/letsencrypt/etc/:/etc/letsencrypt/:ro
            - ~/srv/nginx/etc/:/etc/nginx/:ro
            - ~/srv/nginx/log/:/var/log/nginx/
            - ~/srv/sites/:/sites/:ro
            - ~/srv/sites/default/:/usr/share/nginx/html/:ro

        links:
            - message
 
################################################################################

    message:

        container_name: mosquitto
        image: mosquitto:latest
        restart: always

        ports:
            - 127.0.0.1:1883:1883    # Insecure MQTT restricted to localhost
            - 8883:8883              # Secure MQTT
          # - 9002:9002              # Secure WebSockets - disabled - included as an example for direct WebSockets connections.

        expose:
            - 1883/tcp               # Insecure MQTT       - only available to peer containers
            - 9001/tcp               # Insecure WebSockets - only available to peer containers

        volumes:
            - ~/srv/letsencrypt/etc/:/etc/letsencrypt/:ro
            - ~/srv/mosquitto/etc/:/etc/mosquitto/:ro
            - ~/srv/mosquitto/lib/:/var/lib/mosquitto/
            - ~/srv/mosquitto/log/:/var/log/mosquitto/

        build:
            context: ~/srv/mosquitto/
            args:
                mapURL: https://github.com/jpmens/mosquitto-auth-plug/archive/master.zip
                mongoURL: https://github.com/mongodb/mongo-c-driver/releases/download/1.7.0/mongo-c-driver-1.7.0.tar.gz
 
################################################################################

Build

docker-compose \
    --file ~/srv/docker-compose.yaml \
    build \
    --no-cache \
    ;

Mosquitto

Create paths

mkdir -pv \
    ~/srv/mosquitto/lib/ \
    ~/srv/mosquitto/log/ \
    ;

Initialize

docker run --name tmp-mosquitto-container -d mosquitto:latest

docker cp tmp-mosquitto-container:/etc/mosquitto/ ~/srv/mosquitto/

mv -iv ~/srv/mosquitto/mosquitto ~/srv/mosquitto/etc

docker rm -f tmp-mosquitto-container

echo 'include_dir /etc/mosquitto/conf.d' > ~/srv/mosquitto/etc/mosquitto.conf

for mosquittoConfigFile in \
    'authentication' \
    'authentication-plugin' \
    'bridges' \
    'listeners' \
    'logging' \
    'messaging' \
    'system' \
    ; do
    touch ~/srv/mosquitto/etc/conf.d/${mosquittoConfigFile}.conf
    chmod 640 ~/srv/mosquitto/etc/conf.d/${mosquittoConfigFile}.conf
done

Configure

In the prior step, I created several empty configuration files. Mosquitto carries a conf.d folder for local configuration files which will automatically be parsed if they have a .conf suffix, since Mosquitto's many options can be classified into several categories. You'll want to edit/download each of these files in(to) ~/srv/mosquitto/etc/conf.d/

  • Authentication - basic client security.
  • Authentication Plugin - controls interface to third-party authentication (e.g. if you want to authenticate against a MySQL and/or Redis backend).
  • Bridges - interaction with peer message brokers.
    • Example:


You have devices at your home that aren't powerful enough for encryption (e.g. Arduino), and are connected across a secure connection (Ethernet or WPA/WPA2 Wi-Fi) to a nearby MQTT broker (e.g. a local server).

The communications between Arduino and a local Server would be unencrypted, but the communications between your local server and a remote server would be encrypted, and their communication to each other would be configured as a bridge, so that the remote server can interface with these low-powered devices, without compromising security.

  • Listeners - each and every instance that will be listening, whether it's MQTT or WebSockets.
  • Logging - self-explanatory. :-)
  • Messaging - message handling, such as persistence between service restarts.
  • System - interaction with the container, and if applicable, host machine.

There are many ways that this can go, and I cover a few scenarios in one of my upcoming articles, “Mosquitto Config Cookbook”.
For keeping this article as simple as possible, I'll stick with a single example, spread across these configuration files.

Authentication

authentication.conf
# mosquitto_pyauth
# auth_plugin /usr/local/lib/mosquitto/auth_plugin_pyauth.so
 
# mosquitto-auth-plug
auth_plugin /usr/lib/mosquitto-auth-plugin/auth-plugin.so

Authentication Plugin

You can use mosquitto-auth-plug, mosquitto_pyauth, or if you want to make your own, check out /usr/include/mosquitto_plugin.h which I've included in the Mosquitto Dockerfile via the mosquitto-dev package.

The reason that I didn't compile mosquitto_pyauth for you is because you'll need to choose between Python 2 or Python 3, and you'll probably want to load additional modules into Python. You can use the Dockerfile FROM directive to build upon my Docker Mosquitto image to cover your needs.

For this article, I'm using file-based authentication with the Mosquitto Auth Plugin.

authentication-plugin.conf
auth_opt_backends files
auth_opt_acl_file /etc/mosquitto/acl
auth_opt_password_file /etc/mosquitto/password
auth_opt_anonusername guest
 
# Usernames with this fnmatch(3) (a.k.a glob(3))  pattern are exempt from the module's ACL checking
auth_opt_superusers S*

Example

Initialize files:

for mapFile in \
    'acl' \
    'password' \
    ; do
    touch ~/srv/mosquitto/etc/${mapFile}
    chmod 640 ~/srv/mosquitto/etc/${mapFile}
done

Update ~/srv/mosquitto/etc/acl with usernames and permissions:

user guest
topic read public/#

user adam
topic readwrite world/#

user eve
topic readwrite world/#

Create a password hash:

docker run --rm -it mosquitto np

Update ~/srv/mosquitto/etc/password with a username and generated password hash:

# THIS IS AN EXAMPLE ~/srv/mosquitto/etc/password
SysOp:PBKDF2$sha256$901$0Pwrqzajo66iygqN$khiedWxKasYeOSHtmUB7iRq+Z97i0sLk
adam:PBKDF2$sha256$901$7ycIiFnfzsN+N+3m$9/v159QWJ7ndK5jD9yTfcF2QQ6Jev3Sz
eve:PBKDF2$sha256$901$Si8mm9q0C2ZE6+F3$JS8FpTzIhhnULPyGi1ZYHz9Ei7CNPz4L

Bridges

bridges.conf
# Left blank in this article.  In a subsequent article, I'll discuss bridging.

Listeners

listeners.conf
# MQTT - insecure
listener 1883
protocol mqtt
 
# MQTT - secure
listener 8883
protocol mqtt
cafile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/chain.pem
certfile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/cert.pem
keyfile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/privkey.pem
 
# WebSockets - insecure
listener 9001
protocol websockets
 
# WebSockets - secure - disabled - included as an example for direct WebSockets connections.
# listener 9002
# protocol websockets
# cafile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/chain.pem
# certfile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/cert.pem
# keyfile /etc/letsencrypt/live/YOUR-DOMAIN-GOES-HERE/privkey.pem

Logging

logging.conf
log_dest file /var/log/mosquitto/mosquitto.log
log_type all
websockets_log_level 255
 
connection_messages true
log_timestamp true

Messaging

messaging.conf
persistence true
persistence_location /var/lib/mosquitto/
autosave_interval 1800

System

system.conf
pid_file /var/run/mosquitto.pid
 
# Override permission errors that would be faced with cross-container communication, and multiple owners.
# What is the (best) way to manage permissions for docker shared volumes
# https://stackoverflow.com/a/29799703
user root

Configure Nginx

Remember the template file that I built upon in the prior article? I'll make a slight change to one of the “Available Site” configurations found in ~/srv/nginx/etc/sites-available/, to connect Nginx to Mosquitto's WebSockets for that website.

There are three server{} sections, where as the first two simply exist to forcefully redirect a client's unencrypted requests to a parallel, encrypted side. The encrypted portion is the third one at the bottom. These are the adjustments that I'm going to add in, which will set Nginx to use https://YOUR-DOMAIN-GOES-HERE/ws as the secure WebSockets destination, and pass it to the neighboring Mosquitto container on port 9001 (insecure WebSockets), accessible only from the Nginx container.

You don't need to encrypt communications between the neighboring Nginx and Mosquitto containers, and I need to keep latency as low as possible.

If my topology consisted of running Mosquitto on a different server in a remote location that passed across the Internet in plaintext, then I'd have to encrypt communications between Nginx and Mosquitto.

For the end-user's browser, both the website and WebSockets will be carried over HTTPS (port 443) with the path ( /ws ) being the only visible difference that they will see.

This goes above the third server{} section:

upstream websocket {
	server message:9001;
}

map $http_upgrade $connection_upgrade {
	default upgrade;
	''      close;
}

This goes within the third server{} section:

location /ws {
	proxy_pass http://websocket/ws/;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection $connection_upgrade;

	proxy_connect_timeout       5;
	proxy_send_timeout          5;
	proxy_read_timeout          5;
	send_timeout                5;
}

Using the Site template as an example, the bottom of the configuration file should now be:

# https://%HOSTNAME%/
upstream websocket {
	server message:9001;
}

map $http_upgrade $connection_upgrade {
	default upgrade;
	''      close;
}

server {
 
	location / {
		root   /sites/%HOSTNAME%/public;
		index  index.html index.htm;
	}

	location /ws {
		proxy_pass http://websocket/ws/;
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection $connection_upgrade;

		proxy_connect_timeout       5;
		proxy_send_timeout          5;
		proxy_read_timeout          5;
		send_timeout                5;
	}
 
...

Micro-services

Start

docker-compose \
    --file ~/srv/docker-compose.yaml \
    up \
    -d \
    ;

You can monitor the logs in ~/srv/nginx/log/ and ~/srv/mosquitto/log/, or omit the -d argument above, if you want to watch the services launch.

Once the services launch, you can test your WebSockets connection with the Eclipse Paho JavaScript utility:

You can use Mosquitto's clients to subscribe and publish to the MQTT side:

docker-compose --file ~/srv/docker-compose.yaml \
    exec message \
    mosquitto_sub -h localhost -t "world" -v -u SysOp -P secretPassword
docker-compose --file ~/srv/docker-compose.yaml \
    exec message \
    mosquitto_pub -h localhost -t "world" -m 'Hello, world.' -u SysOp -P secretPassword

Restart

docker-compose \
    --file ~/srv/docker-compose.yaml \
    restart \
    ;

Stop

docker-compose \
    --file ~/srv/docker-compose.yaml \
    down \
    ;

Upgrade

  1. Pull the latest copy of Nginx:
    docker-compose \
        --file ~/srv/docker-compose.yaml \
        pull \
        web \
        ;
  2. Force a new build of Mosquitto:
    docker-compose \
        --file ~/srv/docker-compose.yaml \
        build \
        --no-cache \
        message \
        ;
  3. Stop (and remove) the containers, followed by Starting the containers again.

    The reason that I don't issue a restart is because the containers would simply stop and start. By using the down command listed above, I stop the containers, and remove them. This assists for the next step, where I remove unused containers and images.

  4. Clean-up:
    docker rm $(docker ps --all --quiet --no-trunc --filter 'status=exited');
    docker rmi $(docker images --quiet --filter 'dangling=true');

Conclusion

Okay, you made it to the end of this really long article, but it was worth it! You now have 5 separate services running: 3 of these are exposed to your audience, and 2 are available for your host server and peer containers to use:

  • Secure
    1. Website with mandatory HTTP redirects to HTTPS
    2. WebSockets
    3. MQTT
  • Insecure
    1. Websockets
    2. MQTT

💪🤓💪

You can now push and pull real-time information to clients (both people and machines) across multiple protocols. Where do you go from here? If you're wanting to build a system dashboard or a video game that requires live information, then this is how you do it! If you were going to design a streaming service, it would be a little different, as MQTT and WebSockets both make use of the TCP protocol. In the event that you were wanting to provide streams such as what NPR and C-SPAN provide, you'd want to use the UDP protocol. The new MQTT-SN does in fact support UDP, and perhaps a topic that I'll write about in the future.

All of that said, I have several GUI/Front-end engines that I prefer to use, and they each have their own pros and cons, but all are capable of working (or being shoehorned to work) with MQTT and/or WebSockets:

2017/09/04/tying-mqtt-websockets-and-nginx-together-with-docker.txt · Last modified: 2017/12/31 00:32 by Administrator