Consume records in order

[david-eckardt-sociomantic]

Some applications need the records to be in order. The current disk overflow push strategy, which favours piping records through the memory queue over the disk overflow if possible, can change the order dramatically. However, there are applications that need the current strategy.

sociomantic-tsunami/dmqnode#30 discusses how to implement a DMQ channel where a Consumer pops the records strictly in the order in which they were pushed. Important for the node/client protocol is that the Consume request needs to be extended to work as follows.
We add a new preserve_order parameter to the Consume request. This parameter specifies whether

• the memory queue has priority (the current strategy) (false) or
• all records in this channel should stay in order (true).

The first Consume request that connects to a certain channel with a certain subscriber name sets the preserve_order. From this point on the option is permanently set for this channel/subscriber combination. All subsequent Consume requests with the same channel/subscriber combination are expected to specify the same value for preserve_order as the first one did, otherwise the Consume request will fail. This seems feasible because multiple consumers for the same channel and subscriber are expected to be instances of the same application.

Should there be a need to change the preserve_order option for an existing channel, it would be possible to allow it as long as there is no disk overflow for the channel.

As there is no need for it, the Pop request will not be extended. Once a channel has been consumed from Pop requests are not allowed for it anyway.

So the new protocol features would be

• dmqproto.client.request.Consume
  • Args: additional bool preserve_order field
  • Notification: additional NodeInfo field to report a preserve_order conflict error
• dmqproto.common.Consume
  • New message type to report a preserve_order conflict error