Concurrency/fault-tolerance for aggregated data

[danielhoshi]

Hi!

We would like to use Vector to collect logs from Kafka topics and generate aggregations with some different time windows (e.g. counts/sums for 10 min, 1 hour, 1 day, etc). We were planning to store the results on an external storage like Redis. Some questions that we have:

• Is it possible to run parallel Vector instances to do these aggregations? Do I need something like multiple agents + central aggregator?
• Regarding fault-tolerance, if a Vector instance crashes, do we lose current track of aggregated data? Would reducing the sink intervals (like every 10 seconds) be a possible solution for this or it would probably impact a lot the performance?

[pront]

Hi @danielhoshi, on fault tolerance, the following resources are relevant here:

• https://vector.dev/docs/about/under-the-hood/architecture/buffering-model/
• https://vector.dev/docs/about/under-the-hood/architecture/end-to-end-acknowledgements/

Is it possible to run parallel Vector instances to do these aggregations? Do I need something like multiple agents + central aggregator?

Yes. Although I lack the full context on how you intend to shard your data, I don't see any blockers. As always, I would start with the simplest setup possible, stress test it, and only if it breaks I would consider a more distributed solution.

[danielhoshi]

On my tests with acknowledgements enabled, the consumer group offsets were only updated after a new message arrived on Kafka (and not after the event was delivered to sink, like I was expecting). Do you know why this happens? Is it expected?

I am using a simple console sink like this:

sinks:
  console:
    acknowledgements:
      enabled: true
    inputs:
      - aggregate_custom
    type: console
    encoding:
      codec: json