MQTT library direction

[tve]

I've started to use the new mqtt.js library (and its dependency mqttclient.js) in examples/io/tcp/mqtt* and there's a lot to like! I'm curious about where this API leads and thus what Moddable's plans are. Specifically:

• it's great that QoS 1 is supported (and 2 as well, but that's not for me)
• it's great that it emulates the popular Node.js MQTT.js library
• it's great that it's written with embedded system constraints in mind

What concerns me is resource consumption. Specifically:

• Publish simply queues outgoing messages if the socket can't take them, unless the producer gates the next message on the "sent/acked" callback this can rapidly lead to an out-of-memory situation. In a system with multiple sensors, each producing its own messages, a brief wifi interruption can lead to interesting memory spikes even if every sensor ensures it has only one message outstanding.
• Receive simply calls a callback. I believe the next callback doesn't happen until the previous one completes (it's quite a few layers to trace to ascertain that) however this is not always sufficient to exert back-pressure. I'd like to send mods over MQTT, which means the callback has to initiate flash writes, which are slow and asynchronous. There seems to be no mechanism to avoid getting overrun by incoming messages. (Reconnecting when there are a pile of messages queued in the server can also be interesting.)

Some possible thoughts:

• It's cool to implement the MQTT.js interface, but perhaps it's not well suited to embedded systems and Moddable (or someone else) should build an interface that is better aligned.
• The resource consumption by publish can be kept in check by putting another layer on top, so just do that.
• The message flood on receive can be tamed by layering a flow-control protocol on top of MQTT, so just do that.
• MQTT 5 (I believe) has some flow-control mechanisms (like the "receive maximum") but because mqttclient immediately ACKs incoming packets that may not be helpful as the stack stands (I don't have MQTT 5 experience)

So... my question really is what Moddable's plans are WRT MQTT. I can (a) work around the issues, (b) try to "enhance" mqtt.js/mqttclient.js, or (c) write my own stack. Having some idea about the plans might help align efforts. Thanks much!

[phoddie]

Thanks for the thoughtful feedback. Glad you are finding value in the MQTT support. Before addressing your specific points, it could be useful to state the goals of the two layers that are there:

• mqttclient is an implementation of the forthcoming ECMA-419 2nd Edition Standard. The design is very lightweight while giving the scripts using it access to all of MQTT 3.1.1 with memory/buffering efficiency. It is minimal by design. We expect that developers will build up from here to provide APIs which are optimal for their specific needs. No single API will work for everyone, especially in embedded where implementation size matters too. Which brings us to...
• MQTT.js is an emulation of a popular library from the Node.js ecosystem. It is built on mqtclient. We think it provides a reasonable API for many embedded projects and that having compatibility with an existing API is valuable to many embedded developers.

Our MQTT.js implementation is not a full emulation of the Node.js version and never will be. We took a slice through the features that made sense at this point. That likely will expand over time based on needs.

Regarding buffer management, let's take send and receive in turn.

For receive, I'm not sure there is a problem for your scenario. When you receive the callback, you can do whatever you like -- including writing to flash. That will block the task so no additional callback will be invoked until that write is complete. We do this for large OTA firmware updates over HTTPS, for example, and it works well. The writes are small enough that they do not block for a meaningful amount of time. (FWIW – it would be straightforward to add a "pause receive" capability to MQTT.js but that would be incompatible with the Node version, which would be preferable to avoid).

For send, there is a mechanism in the full MQTT.js that looks like it could work. The pending messages are put in a Store (see outgoingStore) which the user of the library has access to. If my reading is correct that can be used, for example, to ensure that a sensor reading waiting to be sent can be updated rather than adding another to the send queue. Or it could be used to monitor the number of pending message sends and the total data size of those pending messages. There's some work to explore that but it looks possible within the existing API.