Failed buffer write error in kubernetes_events input plugin

[Akila-I]

Bug Report

We are using kubernetes_events input plugin and forwarding events to OpenSearch via Fluent-Bit (version 3.16). I'm getting this error in fluent-bit pod logs continuously and the pod crashes at some point. I think its because too many events are generated in the cluster and the ring buffer size (default 512 entries) in the input plugin is overflowed. Can we have a configurable to set the ring buffer size for kubernetes_events input plugin or another solution for stop fluent-bit dropping out the events in the cluster?

To Reproduce

• Example log message if applicable:

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=0

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=1

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=2

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=3

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=4

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=5

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=6

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=7

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=8

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=9

[2025/01/08 08:45:00] [error] [input:kubernetes_events:kubernetes_events.0] could not enqueue records into the ring buffer

• Steps to reproduce the problem:
  This behavior is observed in a large k8s cluster with k8s events are generated in a very high frequency.

Expected behavior

All the generated k8s events should be entered to the fluent-bit pipeline through kubernetes_events input.

Screenshots

Your Environment

• Version used: 3.1.6
• Configuration:

        name kubernetes_events
        kube_url https://kubernetes.default.svc/
        tag k8s_events
        tls.verify Off

• Environment name and version (e.g. Kubernetes? What version?): AKS with kubernetes 1.30.4
• Server type and version:
• Operating System and version:
• Filters and plugins: kubernetes_events input plugin

Additional context

Related discussion: #9812