[question] How to create aedes broker cluster in Docker?

[tukusejssirs]

First asked on SO.

---

I need to create a MQTT broker cluster. I want to use aedes (as I already use it, albeit without cluster) and Docker. Note that I have never created a cluster before (with nor without Docker).

Here I found official example how to create a cluster using aedes, but I was thinking about using aedes-cli Docker image.

Now, I have no idea which is better: should I create separate containers per broker (similarly to this question) or use mqemitter-redis/mqemitter-mongodb (as the aedes-cli docs suggest).

As for creating separate containers, I have no idea how to connect them.

As for using mqemitter-redis/mqemitter-mongodb, I have no idea how to setup them in the aedes-cli config file.

I have no need for data persistance accross broker cluster restarts. All I need is to distribute the messages accross multiple brokers in order to improve the runtime performance.

---

Below is a working, single-broker configuration.

• aedes.yml:

version: '3.7'
services:
  aedes:
    container_name: aedes
    image: moscajs/aedes:latest
    restart: always
    stop_signal: SIGINT
    network_mode: host
    command: -c /data/dockerConfig.js
    volumes:
      - /some/path/mqtt:/data
  mongo:
    container_name: mongo
    restart: always
    network_mode: host
    logging:
      driver: none
    image: mvertes/alpine-mongo
    volumes:
      - mongo_data:/data/db
    ports:
      - 27017:27017
volumes:
  mongo_data:
    name: mongo_data

• /some/path/mqtt/dockerConfig.js

module.exports = {
  protos: ['tcp', 'ws'],
  host: 'localhost',
  port: 1883,
  wsPort: 3000,
  wssPort: 4000,
  tlsPort: 8883,
  key: null,
  cert: null,
  rejectUnauthorized: true,
  credentials: null,
  brokerId: 'aedes',
  concurrency: 100,
  queueLimit: 42,
  maxClientsIdLength: 23,
  heartbeatInterval: 60000,
  connectTimeout: 30000,
  stats: false,
  statsInterval: 5000,
  mq: {
    name: 'mongodb',
    options: {
      url: 'mongodb://localhost:27017/aedes'
    }
  },
  verbose: false,
  veryVerbose: false,
  noPretty: false
}

[robertsLando]

the configuration you used is correct but you also need to setup mongodb/redis persistence. mqemitter is used to share messages between aedes instances and mongo to persist qos > 0, retained messages and subscriptions so you also need that.

[tukusejssirs]

@robertsLando, thanks for reply! 🙏

I don’t need persistence nor QOS > 0. Do I still need to setup persistence? All I need to have a cluster and to enable the communication between cluster nodes/brokers.

As for the main question: as I understand it, the config in the OP is not yet a cluster config, is it? If it is not a cluster yet, how to make a cluster config?

• Should I create a Docker container per cluster node/broker? How could I make the brokers communicate with each other?
• If I should go for config like this, how can I do this using aedes-cli in Docker container?

[robertsLando]

You should setup both mqemitter and persistence in order to run clusters

Then follow this guide to provide the load balancing:
https://pspdfkit.com/blog/2018/how-to-use-docker-compose-to-run-multiple-instances-of-a-service-in-development/
or
https://sandny.com/2019/11/03/nginx-load-balancer-with-docker-for-nodejs-services/

(there are a lot)

You will have N instances of aedes broker running, mqemitter allows them to share messages (so if you have a client A connected to instance 1 and a client B connected to instance 2 client A will be able receive messages published to instance 2 and vice versa). Persistence is needed to share the same state between all aedes instances

[tukusejssirs]

Thanks again. The linked site help me a lot, however, the ports (port, wsPort, wssPort, tlsPort) collide. Only once container instance runs successfully, others fail with EADDRINUSE (obviously). See below my current config. I think the culprit is that currently I need to use network_mode set to host (all I actually need is to access the broker from on host itself, as my app is not yet running in Docker).

Is there a way to change the ports per container instance? Or how could I work around the issue?

• aedes.yml:

version: '3.7'
services:
  aedes:
    # Note: I needed to remove `container_name`, as each container instance needs to have different name.
    # container_name: aedes
    image: moscajs/aedes:latest
    restart: always
    stop_signal: SIGINT
    deploy:
      # Scale = 3
      replicas: 3
    network_mode: host
    command: -c /data/dockerConfig.js
    volumes:
      - /some/path/mqtt:/data
  mongo:
    container_name: mongo
    restart: always
    network_mode: host
    logging:
      driver: none
    image: mvertes/alpine-mongo
    volumes:
      - mongo_data:/data/db
    ports:
      - 27017:27017
volumes:
  mongo_data:
    name: mongo_data

• /some/path/mqtt/dockerConfig.js

module.exports = {
  protos: ['tcp', 'ws'],
  host: 'localhost',
  port: 1883,
  wsPort: 3000,
  wssPort: 4000,
  tlsPort: 8883,
  key: null,
  cert: null,
  rejectUnauthorized: true,
  credentials: null,
  brokerId: 'aedes',
  concurrency: 100,
  queueLimit: 42,
  maxClientsIdLength: 23,
  heartbeatInterval: 60000,
  connectTimeout: 30000,
  stats: false,
  statsInterval: 5000,
  persistence: {
    name: 'mongodb',
    options: {
      url: 'mongodb://localhost:27017/aedes'
    }
  },
  mq: {
    name: 'mongodb',
    options: {
      url: 'mongodb://localhost:27017/aedes'
    }
  },
  verbose: false,
  veryVerbose: false,
  noPretty: false
}

---

Update

I understand I can append --port to command in Docker compose file, but how can I make the ports different in each container instance? Also I need to use both tcp and ws protocols; how could I define them both use --port and --protos (I think it is not yet possible)? I think I need to do it similarly to this, but I have no idea how to make it. I understand that it is related more to Docker than aedes-cli though.

[robertsLando]

The approach I suggest is to create multiple aedes containers directly in compose file naming them like aedes1 aedes2 and aedes3 ... based on how many instances you need, the configuration they share is the same just but then inside nginx you will have something like:

upstream aedes {
        server aedes1::1883;
        server aedes2::1883;
		server aedes3::1883;
    }

and

server { 
      location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass "http://aedes";
      }
    }

You don't have to to expose aedes or mongodb ports in docker-compose (remove network_mode: host from your docker-compose file).

[tukusejssirs]

@robertsLando, sorry for being away for a while. I am in the middle of dockerising my app, therefore now I came back to this issue.

I can successfully run the following aedes container. According to docker logs aedes, all three replicas should be running (both TCP and WS), however, I cannot connect to it from another container running in the same network (someNetwork). It throws ECONNREFUSED error.

For now, I have not done config in nginx related to aedes and its load-balancing. Should I implement that if I want to access it from within someNetwork only?

What could be wrong with this config?

# docker-compose
aedes:
  image: moscajs/aedes:latest
  restart: unless-stopped
  stop_signal: SIGINT
  deploy:
    replicas: 3
  networks:
    - someNetwork
  command: -c /data/aedes.js
  volumes:
    - /some/path/aedes.js:/data/aedes.js:ro

// /some/path/aedes.js
module.exports = {
  protos: ['tcp', 'ws'],
  host: 'localhost',
  port: 8193,
  wsPort: 4000,
  wssPort: 4001,
  tlsPort: 8883,
  key: null,
  cert: null,
  rejectUnauthorized: false,
  credentials: null,
  brokerId: 'aedes',
  concurrency: 0,
  queueLimit: 200,
  maxClientsIdLength: 23,
  heartbeatInterval: 60000,
  connectTimeout: 30000,
  stats: false,
  statsInterval: 5000,
  persistence: {
    name: 'mongodb',
    options: {
      url: 'mongodb://mongo:27017/aedes'
    }
  },
  mq: {
    name: 'mongodb',
    options: {
      url: 'mongodb://mongo:27017/aedes'
    }
  },
  verbose: false,
  veryVerbose: false,
  noPretty: false
}

// Client (in another container); both TCP and WS connection fails with `ECONNREFUSED`
const {connect} = require('mqtt')
const tcpPort = 8193
const tcpClient = connect(`mqtt://aedes:${tcpPort}`)

tcpClient.on('error', e => {
  console.log(`MQTT error:`, e)
})

const wsPort = 4000
const wsClient = connect(`mqtt://aedes:${wsPort}`)

wsClient.on('error', e => {
  console.log(e)
})

// Sample output:
// - same output for both TCP and WS;
// - IP addresses cycle through the replicas.
Error: connect ECONNREFUSED 172.29.0.6:8193
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1195:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '172.29.0.6',
  port: 8193
}

---

Update: I also tried to run the client code within aedes container using 127.0.0.1, but I still get ECONNREFUSED.

Update 2: When I comment out host: 'localhost' from aedes.js (or when I use host: '0.0.0.0', it works locally in the aedes container, but only for WS, never for TCP. I tried it in all instances of aedes. It still does not work in another container (with and without using a custom Docker network).

Update 3: I believe that the ports are not open outside of the aedes container. I tested this using nmap both in the container of my app and on the host system. I tried to add 8193 and 4000 to ports in the Docker compose file (without mapping host port specification), however, it didn’t solve the issue.

Update 4: I ‘solved’ this issue by modifying the official example. I believe the ports issue is on the Docker image/container level, but I have no idea why it does not work.