Fluentd not able to run as non root

[Tilusch]

Describe the bug:
When running flluentd asnonroot and a diffrent user the statefulset is not coming up. Is there no way to run fluentd as non root when using the logging operator ressources?

Expected behaviour:
Fluentd is able to run as non root and a specified user.

Steps to reproduce the bug:
Create the following logging ressource and access the fluentd pod logs the output should be similiar:

/usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:121:in `initialize': Permission denied @ rb_sysopen - /fluentd/log/out (Errno::EACCES)
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:121:in `open'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:121:in `create_logfile'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:110:in `rescue in open_logfile'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:106:in `open_logfile'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:85:in `set_dev'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:19:in `initialize'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:707:in `new'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:707:in `setup_global_logger'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:624:in `configure'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/fluentd-1.17.1/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'
/usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:108:in `initialize': No such file or directory @ rb_sysopen - /fluentd/log/out (Errno::ENOENT)
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:108:in `open'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:108:in `open_logfile'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:85:in `set_dev'
	from /usr/local/bundle/gems/logger-1.6.3/lib/logger/log_device.rb:19:in `initialize'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:707:in `new'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:707:in `setup_global_logger'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/supervisor.rb:624:in `configure'
	from /usr/local/bundle/gems/fluentd-1.17.1/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/fluentd-1.17.1/bin/fluentd:15:in `<top (required)>'
	from /usr/local/bundle/bin/fluentd:25:in `load'
	from /usr/local/bundle/bin/fluentd:25:in `<main>'

Additional context:

• If tried to add the following capabilities but without success:
• DAC_READ_SEARCH

Environment details:

• Kubernetes version (e.g. v1.15.2): v1.31.2
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): AKS
• logging-operator version (e.g. 2.1.1): 5.0.1
• Install method (e.g. helm or static manifests): Helm
• Logs from the misbehaving component (and any other relevant logs):
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: logging-operator
    app.kubernetes.io/version: 5.0.1
    helm.sh/chart: logging-operator-5.0.1
  name: logging-operator
spec:
  clusterDomain: cluster.local.
  controlNamespace: logging-operator
  enableRecreateWorkloadOnImmutableFieldChange: true
  fluentd:
    bufferVolumeImage:
      pullPolicy: Always
    bufferVolumeMetrics:
      prometheusRules: true
      serviceMonitor: true
    bufferVolumeResources:
      limits:
        cpu: 50m
        memory: 50M
      requests:
        cpu: 1m
        memory: 10M
    configCheckResources:
      limits:
        cpu: 150m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 32Mi
    configReloaderImage:
      pullPolicy: Always
    configReloaderResources:
      limits:
        cpu: 150m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 32Mi
    fluentOutLogrotate:
      age: '10'
      enabled: true
      path: /fluentd/log/out
      size: '10485760'
    image:
      pullPolicy: Always
    livenessProbe:
      exec:
        command:
          - /bin/sh
          - '-c'
          - >
            LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; if [
            ! -e /buffers ]; then
              exit 1;
            fi; touch -d date -d "@$(($(date +%s) -
            $LIVENESS_THRESHOLD_SECONDS))" /tmp/marker-liveness; if [ -z "$(find
            /buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
              exit 1;
            fi;
      initialDelaySeconds: 600
      periodSeconds: 60
    metrics:
      prometheusRules: true
      serviceMonitor: true
    readinessDefaultCheck:
      bufferFileNumber: false
      bufferFreeSpace: true
      bufferFreeSpaceThreshold: 90
      failureThreshold: 1
      initialDelaySeconds: 5
      periodSeconds: 30
      successThreshold: 3
      timeoutSeconds: 3
    scaling: {}
    security:
      podSecurityContext:
        fsGroup: 10013
        fsGroupChangePolicy: Always
        runAsGroup: 10013
        runAsNonRoot: true
        runAsUser: 10013
        supplementalGroups: []
        sysctls: []
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        readOnlyRootFilesystem: false
        seLinuxOptions: {}
        seccompProfile:
          type: RuntimeDefault
        sysctls: []

/kind bug

[csatib02]

Hey @Tilusch,

Did you check whether your config works, when running as root?

[Tilusch]

Hi @csatib02
Indeed the config works when i use the following securityContext instead and run as root:

      podSecurityContext:
        fsGroup: 10013
        fsGroupChangePolicy: Always
        supplementalGroups: []
        sysctls: []
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: false
        seLinuxOptions: {}
        seccompProfile:
          type: RuntimeDefault

[pepov]

The fluentd image is being relocated into the repo. Once we finished that we get back to this.

[sebastiangaiser]

For me the following configuration is working. However the sidecar is causing problems as mentioned:

      fluentd:
        security:
          podSecurityContext:
            fsGroup: 101
            runAsGroup: 101
            runAsNonRoot: true
            runAsUser: 100
            seccompProfile:
              type: RuntimeDefault
          securityContext:
            allowPrivilegeEscalation: false
            # true not working as `buffer-metrics-sidecar sh: /prometheus/buffer-size.sh: Permission denied`
            readOnlyRootFilesystem: false
            capabilities:
              drop:
                - ALL

[csatib02]

@Tilusch

Would it be a viable solution for you if we would allow adding custom init-containers via the logging resource?

[Tilusch]

@csatib02 doesnt it make more sense to edit the buffer image to allow running as non root for complexity reasons, but anyway this would be a possible workaround for me!
@sebastiangaiser I can start the pods exept the buffer-metrics-sidecar with your config. I tried again to set the uid/gid to 10013 but then the config is not working and i get the same error as above. Is it mandatory to use 101 and 100 for fluentd?

[sebastiangaiser]

@Tilusch where did you have these settings (fsGroup: 10013, runAsUser: 10013, ...) from?
I took them from https://kube-logging.dev/docs/logging-infrastructure/security/#security-contexthttpskubernetesiodocstasksconfigure-pod-containersecurity-context and extended the config a bit.

[csatib02]

We are in the process of pulling in dependency images under this repo, we are going to handle the node-exporter next, this will be fixed as part of that.

As for the user and group id's they are set within our custom fluentd image: https://github.com/kube-logging/logging-operator/blob/master/images/fluentd/Dockerfile

[Tilusch]

@sebastiangaiser i made them up myself since its a good practice to use non default users when possible.

[pepov]

@Tilusch yes it should be a good idea to use 101 and 100 since the docker image chown's the files for that user and group explicitly.