Fluentbit in_docker crash

[nitrogene]

Hello,

I have installed td-agent-bit on wsl2/ubuntu, and have troubles to make the in_docker plugin works.

Here's an extract of td-agent-bit.conf

[INPUT]
# https://docs.fluentbit.io/manual/pipeline/inputs/docker-metrics
    Name   docker
    Tag    docker.metrics

[INPUT]
# https://docs.fluentbit.io/manual/pipeline/inputs/docker-events
    Name   docker_events
    Tag    docker.events


[FILTER]
# https://docs.fluentbit.io/manual/pipeline/filters/record-modifier
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}

[OUTPUT]
    Name forward
    Match *
    Host 127.0.0.1
    Port 24224
    tls On
    tls.verify On
    tls.ca_file /etc/certs/graylog/certs/ca.crt.pem
    tls.crt_file /etc/certs/graylog/certs/client.crt.pem
    tls.key_file /etc/certs/graylog/private/client.key.pem
    tls.key_passwd ${TLS_PRIVATE_KEY_PASSPHRASE}
    Shared_Key  ${SHARED_KEY}

And as soon as I start the agent:

Fluent Bit v1.9.0
* Copyright (C) 2015-2021 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/03/16 16:01:55] [ info] [engine] started (pid=7746)
[2022/03/16 16:01:55] [ info] [storage] version=1.1.6, initializing...
[2022/03/16 16:01:55] [ info] [storage] in-memory
[2022/03/16 16:01:55] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/03/16 16:01:55] [ info] [cmetrics] version=0.3.0
[2022/03/16 16:01:55] [ warn] [input:thermal:thermal.4] thermal device file not found
[2022/03/16 16:01:55] [ info] [input:docker_events:docker_events.6] listening for events on /var/run/docker.sock
[2022/03/16 16:01:55] [ info] [sp] stream processor started
[2022/03/16 16:01:56] [ info] [output:forward:forward.0] worker #0 started
[2022/03/16 16:01:56] [ info] [output:forward:forward.0] worker #1 started
[2022/03/16 16:01:56] [error] [plugins/in_docker/docker.c:315 errno=2] No such file or directory
[2022/03/16 16:01:56] [engine] caught signal (SIGSEGV)
[2022/03/16 16:01:56] [error] [input:docker:docker.5] error gathering CPU data from /sys/fs/cgroup/cpu/docker/c6947781249f695c7811ce5da932b2aa79f28ea95b89c0a12b9eb841fa28302e/cpuacct.usage
#0  0x558ead4e587c      in  flush_snapshot() at plugins/in_docker/docker.c:701
#1  0x558ead4e5a03      in  flush_snapshots() at plugins/in_docker/docker.c:728
#2  0x558ead4e5c21      in  cb_docker_collect() at plugins/in_docker/docker.c:798
#3  0x558ead482141      in  flb_input_collector_fd() at src/flb_input.c:1203
#4  0x558ead4986da      in  flb_engine_handle_event() at src/flb_engine.c:439
#5  0x558ead4986da      in  flb_engine_start() at src/flb_engine.c:761
#6  0x558ead474023      in  flb_lib_worker() at src/flb_lib.c:626
#7  0x7f237eba2608      in  ???() at ???:0
#8  0x7f237e4cb162      in  ???() at ???:0
#9  0xffffffffffffffff  in  ???() at ???:0

If I deactivate the in_docker plugin by commenting the relevant lines in the configuration file, it just works => I can see the host metrics (cpu, mem, etc..).

Here's the content of the content of the /sys/fs/cgroup/cpu/docker/c6947781249f695c7811ce5da932b2aa79f28ea95b89c0a12b9eb841fa28302e/ folder :

$ ls /sys/fs/cgroup/cpu/docker/c6947781249f695c7811ce5da932b2aa79f28ea95b89c0a12b9eb841fa28302e/
cgroup.clone_children  cpu.cfs_period_us  cpu.rt_period_us   cpu.shares  notify_on_release
cgroup.procs           cpu.cfs_quota_us   cpu.rt_runtime_us  cpu.stat    tasks

Any idea ?

[agup006]

My guess is that the API calls we used for in_docker need to be updated with the later versions. Perhaps we should update entirely to Prometheus based metrics from in_docker.

You can use the Prometheus Scraper (https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-scrape-metrics) + Docker Prometheus metrics to achieve this today (https://docs.docker.com/config/daemon/prometheus/)

[nitrogene]

Thank, I'll take a look. For the record, this discussion was turned into an issue