Consumer Offsets, support via internal topic/partition

[markpapadakis]

According to @miguno ( I need to go through the Kafka codebase to figure out what they do exactly ), they are maintaining an internal/special-purpose topic __consumer_offsets, where the policy is configured as cleanup, because they only really care for the last value, and, presumably, on startup, the partition is replayed in order to restore the state.

That's a clever and pragmatic way to implement it.

See also: http://www.slideshare.net/jjkoshy/offset-management-in-kafka

[markpapadakis]

See: https://www.ctheu.com/2017/08/07/looking-at-kafka-s-consumers-offsets/

A new topic_partition::foreach_msg() has been implemented and can be used to effortlessly iterate across all messages, both in the RO segments and the current RW segment. This can be used to materialize state by iterating all messages in, say, __consumer_offsets partitions.