fluentbit pods stops after few seconds - need help resolving the issue

[mali-ewl]

fluentbit daemonset on GCP GKE stops after few seconds. There are no errors or warning in the debug logs. Any help would be appretiated.

[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] scan_glob add(): /var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_app-config-copy-from-image-286c121669a48af46cfa4432c67a82413e6e147074903eff9091b322f9e86441.log, inode 4981
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] inode=5092 with offset=28203 appended as /var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_myapp-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] scan_glob add(): /var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_myapp-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log, inode 5092
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] 3 new files found on path '/var/log/containers/*.log'
[2022/03/14 14:10:38] [debug] [tail:gvisor] created event channels: read=35 write=36
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] flb_tail_fs_inotify_init() initializing inotify tail input
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] inotify watch fd=43
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] scanning path /var/log/pods/*/*.log
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] cannot read info from: /var/log/pods/*/*.log
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] 0 new files found on path '/var/log/pods/*/*.log'
[2022/03/14 14:10:38] [debug] [filter:modify:modify.1] Initialized modify filter with 0 conditions and 1 rules
[2022/03/14 14:10:38] [debug] [filter:modify:modify.3] Initialized modify filter with 0 conditions and 1 rules
[2022/03/14 14:10:38] [debug] [stackdriver:stackdriver.0] created event channels: read=45 write=46
[2022/03/14 14:10:38] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2022/03/14 14:10:38] [debug] [output:stackdriver:stackdriver.0] JWT signature: ......
[2022/03/14 14:10:38] [debug] [http_client] not using http_proxy for header
[2022/03/14 14:10:38] [ info] [oauth2] HTTP Status=200
[2022/03/14 14:10:38] [debug] [oauth2] payload:
{"access_token":"......","expires_in":3599,"token_type":"Bearer"}
[2022/03/14 14:10:38] [ info] [oauth2] access token from 'www.googleapis.com:443' retrieved
[2022/03/14 14:10:38] [debug] [upstream] KA connection #44 to www.googleapis.com:443 is now available
[2022/03/14 14:10:38] [debug] [router] match rule tail.0:stackdriver.0
[2022/03/14 14:10:38] [debug] [router] match rule tail.1:stackdriver.0
[2022/03/14 14:10:38] [ info] [output:stackdriver:stackdriver.0] worker #1 started
[2022/03/14 14:10:38] [ info] [output:stackdriver:stackdriver.0] worker #0 started
[2022/03/14 14:10:38] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/03/14 14:10:38] [ info] [sp] stream processor started
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] inode=527013 file=/var/log/containers/db-dbeaver-7ccdd479cd-66fjw_default_myapp-poc-db-04ad6cdbd47f1c9cd5991d4bde8afbd29a4beb094e89284cfa260c2acb5a58ee.log promote to TAIL_EVENT
[2022/03/14 14:10:38] [ info] [input:tail:kube_containers] inotify_fs_add(): inode=527013 watch_fd=1 name=/var/log/containers/db-dbeaver-7ccdd479cd-66fjw_default_myapp-poc-db-04ad6cdbd47f1c9cd5991d4bde8afbd29a4beb094e89284cfa260c2acb5a58ee.log
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] inode=4981 file=/var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_app-config-copy-from-image-286c121669a48af46cfa4432c67a82413e6e147074903eff9091b322f9e86441.log promote to TAIL_EVENT
[2022/03/14 14:10:38] [ info] [input:tail:kube_containers] inotify_fs_add(): inode=4981 watch_fd=2 name=/var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_app-config-copy-from-image-286c121669a48af46cfa4432c67a82413e6e147074903eff9091b322f9e86441.log
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] inode=5092 file=/var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_myapp-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log promote to TAIL_EVENT
[2022/03/14 14:10:38] [ info] [input:tail:kube_containers] inotify_fs_add(): inode=5092 watch_fd=3 name=/var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_myapp-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log
[2022/03/14 14:10:38] [debug] [input:tail:kube_containers] [static files] processed 0b, done
[2022/03/14 14:10:38] [debug] [input:tail:gvisor] [static files] processed 0b, done
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] scanning path /var/log/containers/*.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] scan_blog add(): dismissed: /var/log/containers/db-dbeaver-7ccdd479cd-66fjw_default_myapp-poc-db-04ad6cdbd47f1c9cd5991d4bde8afbd29a4beb094e89284cfa260c2acb5a58ee.log, inode 527013
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/fluentbit-gke-262w9_kube-system_fluentbit-b36729a96859f03b529da116b6d4e6d10103117e7e06bb6c27479206bb84ba00.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/fluentbit-gke-262w9_kube-system_fluentbit-gke-87ed7613a86cca178c0300d6ee43833eef3aaa0ac92b630b0d338f1c76eb9134.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/fluentbit-gke-custom-ds-v7xjr_kube-system_fluentbit-custom-0f26f46735e7f3aa895636dda7f25f973e65c102d2154b6f57811435f95f59b1.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/gke-metrics-agent-wc555_kube-system_gke-metrics-agent-96261c08f35bec91c75713a741d266d9561b57366a20c85a81617e8bc514e516.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/kube-dns-697dc8fc8b-mpm89_kube-system_dnsmasq-ec4bd2e94ddb710d14e365aabdf31a8735554303be4caef6082eecd33c8d98a4.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/kube-dns-697dc8fc8b-mpm89_kube-system_kubedns-c84e286474331a24e1d85652fc82115ab3baf26d0a3efb66f5043a12d82ec781.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/kube-dns-697dc8fc8b-mpm89_kube-system_prometheus-to-sd-771af07cdf2eeff343be2675a5921ed128dc8f77994f370b7cad1b0283c43106.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/kube-dns-697dc8fc8b-mpm89_kube-system_sidecar-8fad391cf1372ca7e77a587b11c4427be66ed10e93c5cf0d4233ad05ce00a96d.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/kube-proxy-gke-test-myapp-cluste-test-myapp-cluste-b360fe0f-1g9s_kube-system_kube-proxy-3eab35f18d47756a5cb5f27f55d52b1e93e6865b5a4164d3df0351828d111276.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/pdcsi-node-nxvnd_kube-system_csi-driver-registrar-f00ab57e9ac2e1795c45327d0c0f2afcb5691a497cd6382a449d9e8e979c4e48.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/pdcsi-node-nxvnd_kube-system_gce-pd-driver-2f596d19979b3417a29e3fbfeff57e6670b8e83e78ceb36e821995a77ed54687.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/workload-metrics-fp9nv_kube-system_collector-680e4377399fdeda5434a06f834706498ad32f0a9c66f429d784490e5dad11d6.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/workload-metrics-fp9nv_kube-system_collector-bf7919abe48583375943a6a65226fca980aa1eaef203219c6162d5ad67591266.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] excluded=/var/log/containers/workload-metrics-fp9nv_kube-system_configmap-reloader-f936bc78c1ef3ede8e1c84bdac23de0af73a3b26d39120bd3cc68bdee96b08b5.log
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] scan_blog add(): dismissed: /var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_app-config-copy-from-image-286c121669a48af46cfa4432c67a82413e6e147074903eff9091b322f9e86441.log, inode 4981
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] scan_blog add(): dismissed: /var/log/containers/myapp-backoffice-59764f5f86-sngl5_default_myapp-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log, inode 5092
[2022/03/14 14:10:39] [debug] [input:tail:kube_containers] 0 new files found on path '/var/log/containers/*.log'
....
....
[2022/03/14 14:11:11] [debug] [input:tail:kube_containers] scan_blog add(): dismissed: /var/log/containers/xs2a-backoffice-59764f5f86-sngl5_default_xs2a-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log, inode 5092
[2022/03/14 14:11:11] [debug] [input:tail:kube_containers] 0 new files found on path '/var/log/containers/*.log'
[2022/03/14 14:11:11] [ warn] [engine] service will shutdown in max 5 seconds
[2022/03/14 14:11:12] [ info] [engine] service has stopped (0 pending tasks)
[2022/03/14 14:11:12] [debug] [input:tail:kube_containers] inode=527013 removing file name /var/log/containers/db-dbeaver-7ccdd479cd-66fjw_default_xs2a-poc-db-04ad6cdbd47f1c9cd5991d4bde8afbd29a4beb094e89284cfa260c2acb5a58ee.log
[2022/03/14 14:11:12] [ info] [input:tail:kube_containers] inotify_fs_remove(): inode=527013 watch_fd=1
[2022/03/14 14:11:12] [debug] [input:tail:kube_containers] inode=4981 removing file name /var/log/containers/xs2a-backoffice-59764f5f86-sngl5_default_app-config-copy-from-image-286c121669a48af46cfa4432c67a82413e6e147074903eff9091b322f9e86441.log
[2022/03/14 14:11:12] [ info] [input:tail:kube_containers] inotify_fs_remove(): inode=4981 watch_fd=2
[2022/03/14 14:11:12] [debug] [input:tail:kube_containers] inode=5092 removing file name /var/log/containers/xs2a-backoffice-59764f5f86-sngl5_default_xs2a-backoffice-9d83039eed6998ded586ff7e5eea243cb50e332741fcecdce50d3b11276452d9.log
[2022/03/14 14:11:12] [ info] [input:tail:kube_containers] inotify_fs_remove(): inode=5092 watch_fd=3
[2022/03/14 14:11:12] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopping...
[2022/03/14 14:11:12] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopped
[2022/03/14 14:11:12] [ info] [output:stackdriver:stackdriver.0] thread worker #1 stopping...
[2022/03/14 14:11:12] [ info] [output:stackdriver:stackdriver.0] thread worker #1 stopped

fluent configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentbit-gke-custom-cm
  namespace: kube-system
  labels:
    k8s-app: fluent-bit
data:
  #Configuration files: server, input, filters, and output
  #=======================================================
  fluent-bit.conf: |-
    [SERVICE]
        Flush         2
        #Grace         5
        Log_Level     trace
        Log_File      /var/log/fluentbit_custom.log
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    [INPUT]
        Name            tail
        Alias           kube_containers
        Tag             kube.<namespace_name>.<pod_name>.<container_name>
        Tag_Regex       var.log.containers.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
        Path             /var/log/containers/*.log
        Exclude_Path     /var/log/containers/*_kube-system_*.log,/var/log/containers/*_logging_*.log,/var/log/containers/*_ingress-nginx_*.log,/var/log/containers/*_kube-node-lease_*.log,/var/log/containers/*_kube-public_*.log,/var/log/containers/*_cert-manager_*.log,/var/log/containers/*_prometheus-operator_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log      
        DB               /var/run/flb_kube_custom.db
        Buffer_Max_Size  10MB
        Mem_Buf_Limit    20MB
        Skip_Long_Lines  On
        Refresh_Interval 1
    [FILTER]
        Name         parser
        Match        kube.*
        Key_Name     log
        Reserve_Data True
        Parser       docker
        Parser       containerd
    #This input is used  to watch changes to Kubernetes pod log files live in the
    #directory /var/log/pods/NAMESPACE_NAME_UID. The file name is used to
    #capture the pod namespace, name and runtime name.
    [INPUT]
        Name             tail
        Alias            gvisor
        Tag              kube-pod.<namespace_name>.<pod_name>.<runtime>
        Tag_Regex        \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
        Exclude_Path     /var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*
        Path             /var/log/pods/*/*.log
        DB               /var/run/gvisor_custom.db
        Buffer_Max_Size  10MB
        Mem_Buf_Limit    20MB
        Skip_Long_Lines  On
        Refresh_Interval 5
    [FILTER]
        Name        modify
        Match       *
        Hard_rename log message
    [FILTER]
        Name         parser
        Match        kube_*
        Key_Name     message
        Reserve_Data True
        Parser       glog
        Parser       json
    # level is a common synonym for severity,
    # the default field name in libraries such as GoLang's zap.
    # populate severity with level, if severity does not exist.
    [FILTER]
        Name        modify
        Match       kube_*
        Copy        level severity
    [OUTPUT]
        # write the log records that still have the 'kube.*' tags to Cloud Logging
        Name         stackdriver
        Match        *
        severity_key severity
        tag_prefix   kube.
        # The following fields are necessary. They allow filtering in Cloud Logging
        # based on resource types. Change them accordingly based on your setup.
        k8s_cluster_name test-xs2a-cluster
        k8s_cluster_location  europe-west1
  parsers.conf: |-
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    [PARSER]
        Name        containerd
        Format      regex
        Regex       ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    [PARSER]
        Name        json
        Format      json
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
    [PARSER]
        Name        glog
        Format      regex
        Regex       ^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source_file>[^ \]]+)\:(?<source_line>\d+)\]\s(?<message>.*)$
        Time_Key    time
        Time_Format %m%d %H:%M:%S.%L%z
    [PARSER]
        Name        network-log
        Format      json
        Time_Key    timestamp
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[patrick-stephens]

I'd check what K8S events are being reported to see if something else is killing your pods.

https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application-introspection/

If not, what version, etc are you using?