Missing messages in consumer

[psamim]

Hi,
I have a simple consumer script exactly the same as the one in the readme. I put numbers from 1 to 5 into a topic using a producer. I can consume and see these numbers from 1 to 5 using the kafka-console-consumer.sh, but using poseidon I receive the messages like below.

1
3
5

Any suggestions?

[damien]

Do you have multiple Kafka brokers/partitions? Could be you're just consuming messages from a single partition.

[psamim]

@damien Thanks for the response.

I did not set a partitioner for the producer so it seems that it is publishing messages to more than one partition. Can I have a consumer to consume all messages from a topic regardless of the partition? Or at least a way to know how many partitions are there for a topic.

Thanks

[damien]

Unfortunately, consuming messages from Kafka is quite a bit more complicated than publishing messages to Kafka. I'm in the process of figuring out a similar problem myself and the best ruby based consumer library I've been able to find for the problem is poseidon_cluster.

Setting that up and pointing it at your brokers should ensure you get all the messages you're looking for; otherwise you probably really are seeing a bug in poseidon itself.

[psamim]

I did a workaround which I think is wrong in general. I set the partitioner so that all messages are published to one partition. Then I create a partition consumer for that partition.

[damien]

That does seem to verify that you were only listening to one of multiple partitions. Definitely check out poseidon_cluster; your workaround may fix your problem but you'll be giving up a lot of Kafka's guarantees around persistence/availability if you're running off of a single partition.