Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random kafka partitioning is not working #243

Closed
simPod opened this issue Dec 1, 2023 · 5 comments
Closed

Random kafka partitioning is not working #243

simPod opened this issue Dec 1, 2023 · 5 comments
Labels
bug Something isn't working transport Related to transport libraries (Kafka)

Comments

@simPod
Copy link
Contributor

simPod commented Dec 1, 2023

Describe the bug

Currently, goflow produces to single partition. I tried enabling hashing in order to achieve random partitioning but it seems to have no effect.

To Reproduce

-transport.kafka.hashing=true

(

 -transport.kafka.brokers=... \
 -transport.kafka.version=3.5.0 \
 -transport.kafka.hashing=1 \
 -format=bin \
 -listen 'sflow://:1234?count=16'

)

kafkaConfig.Producer.Partitioner = sarama.NewHashPartitioner

But maybe the key is not nil by default? 🤔

https://github.com/IBM/sarama/blob/d09a287f30bdf73b1c9f53c3f7a4b2ecc2bd41d3/partitioner.go#L171-L174

Expected behavior

Messages produces to all partitions in the topic randomly

GoFlow2:

  • v2.0.0
  • debian pkg
  • OS: linux debian
@simPod simPod added the bug Something isn't working label Dec 1, 2023
@simPod
Copy link
Contributor Author

simPod commented Dec 1, 2023

It also seems setting transport.kafka.hashing=1 is redundant as the NewHashPartitioner is default
image

@lspgn
Copy link
Member

lspgn commented Dec 2, 2023

Thanks for the report! will have a look

@LeoAquarius
Copy link

May be the type ProtoProducerMessage struct function baseKey(h hash.Hash) had m.formatter == nil || len(m.formatter.key) == 0.

@lspgn
Copy link
Member

lspgn commented Dec 4, 2023

Hoping to fix with #246

@lspgn lspgn added the transport Related to transport libraries (Kafka) label Dec 6, 2023
@lspgn
Copy link
Member

lspgn commented Dec 6, 2023

Should be fixed. Let me know if still having the issue.

@lspgn lspgn closed this as completed Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working transport Related to transport libraries (Kafka)
Projects
None yet
Development

No branches or pull requests

3 participants