SELinux super_t label not applied based on manifest seLinuxOptions container config

[btiernay]

I'm attempting to run kubernetes/node-problem-detector on EKS as a DaemonSet. However, the specified seLinuxOptions labels in my manifest don't appear to be applying when created by the Kubelet:

kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
        - name: node-problem-detector
          securityContext:
            privileged: true
            seLinuxOptions:
              user: system_u
              role: system_r
              type: super_t
              level: s0

When I drop into sheltie on the node, I see:

ps -efZ | grep /node-problem-detector

system_u:system_r:control_t:s0-s0:c0.c1023 root 2893721 2893563  0 01:21 ? 00:00:00 /node-problem-detector --logtostderr --config.system-log-monitor=/config/kernel-monitor.json --config.custom-plugin-monitor=/custom-config/health-checker-kubelet.json,/custom-config/health-checker-containerd.json --prometheus-address=0.0.0.0 --prometheus-port=20257 --k8s-exporter-heartbeat-period=5m0s

Notably here is the labeling of system_u:system_r:control_t:s0-s0:c0.c1023. Based on the docs, I would have assumed this would have been system_u:system_r:super_t:s0-s0 or similar.

The control_t and super_t labels allow writes to the API socket. The super_t label allows modifications to any file or directory on the host OS.

Some orchestrators allow SELinux labels to be defined in the container specification, including Kubernetes and Amazon ECS. If control_t or super_t is specified in this way, it will override the default transition rules and the container will run with additional privileges.

This is causing issues when the container goes to access a resource labeled as follows:

journalctl -xef | grep selinux

May 31 19:46:42 ip-10-140-33-86.ec2.internal systemd[1]: selinux: avc:  denied  { status } for auid=n/a uid=0 gid=0 path="/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service" cmdline="" function="mac_selinux_filter" scontext=system_u:system_r:control_t:s0-s0:c0.c1023 tcontext=system_u:object_r:os_t:s0 tclass=service permissive=0

Said resource:

ls -lZ /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service

-rw-r--r--. 1 root root system_u:object_r:os_t:s0 519 May  9 23:45 /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service

Version info:

Details

sestatus

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             fortified
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     denied
Memory protection checking:     actual (secure)
Max kernel policy version:      33

apiclient get

{
  "os": {
    "arch": "x86_64",
    "build_id": "9cd59298",
    "pretty_name": "Bottlerocket OS 1.14.0 (aws-k8s-1.24)",
    "variant_id": "aws-k8s-1.24",
    "version_id": "1.14.0"
  },

Is there anything in particular I am overlooking? Thank you community for any guidance and insight you can afford! 🙇

[bcressey]

privileged: true causes the "what label should I use?" logic in containerd-cri to take a different path, which leads to no specific label being set and getting the default domain transition to control_t.

It's almost possible to roll your own privileged: true equivalent (by adding all capabilities, setting a privileged SELinux label, disabling seccomp).

The main privilege that can't be picked up that way is the ability to modify all devices, since there's no field for that in the security context.

[btiernay]

Thanks @bcressey, that was the missing bit for me!

For the benefit of others, here is the minimal version of the configuration required to get this to work with Bottlerocket:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
    spec:
        env:
        - name: SYSTEMD_IGNORE_CHROOT
          value: "1"
        securityContext:
          privileged: false
          seLinuxOptions:
            level: s0
            role: system_r
            type: super_t
            user: system_u
      hostPID: true
        volumeMounts:
        - mountPath: /dev/kmsg
          name: kmsg
          readOnly: true
        - mountPath: /etc/machine-id
          name: machine-id
          readOnly: true
        - mountPath: /run/systemd
          name: systemd
        - mountPath: /var/run/dbus/
          name: dbus
        - mountPath: /var/run/containerd
          name: containerd
          readOnly: true
        - mountPath: /var/log/
          name: log
          readOnly: true
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /custom-config
          name: custom-config
          readOnly: true
      volumes:
      - hostPath:
          path: /dev/kmsg
        name: kmsg
      - hostPath:
          path: /etc/machine-id
          type: File
        name: machine-id
      - hostPath:
          path: /run/systemd/
          type: ""
        name: systemd
      - hostPath:
          path: /var/run/dbus/
          type: ""
        name: dbus
      - hostPath:
          path: /var/run/containerd
          type: ""
        name: containerd
      - hostPath:
          path: /var/log/
        name: log
      - hostPath:
          path: /etc/localtime
          type: FileOrCreate

Hopefully this can be improved upon in what comes out of #3157, but this at least unblocks me for now. Much appreciated! ❤️

[btiernay]

@bcressey Coming back to this, I noticed that NPD was actually having permission issues trying to read /dev/kmsg without privileged: true, but that breaks the SELinux labelling as discussed. Do you have any suggestions on how that might be achieved?

Details

Liberal manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/instance: node-problem-detector
    app.kubernetes.io/name: node-problem-detector
  name: node-problem-detector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-problem-detector
      app.kubernetes.io/instance: node-problem-detector
      app.kubernetes.io/name: node-problem-detector
  template:
    metadata:
      annotations:
        checksum/config: 1f83465a5e89cdccb83f5984a12db88ac5df3f2b297832d237bf9e4cd21c1a5d
        prometheus.io/port: "20257"
        prometheus.io/scrape: "true"
      labels:
        app: node-problem-detector
        app.kubernetes.io/instance: node-problem-detector
        app.kubernetes.io/name: node-problem-detector
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - 'exec /node-problem-detector --logtostderr --config.system-log-monitor=/config/kernel-monitor.json
          --config.custom-plugin-monitor=/custom-config/health-checker-kubelet.json,/custom-config/health-checker-containerd.json
          --prometheus-address=0.0.0.0 --prometheus-port=20257 --k8s-exporter-heartbeat-period=5m0s  '
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: SYSTEMD_IGNORE_CHROOT
          value: "1"
        image: <snip>
        imagePullPolicy: IfNotPresent
        name: node-problem-detector
        ports:
        - containerPort: 20257
          name: exporter
        resources:
          limits:
            cpu: 10m
            memory: 80Mi
          requests:
            cpu: 10m
            memory: 80Mi
        securityContext:
          capabilities:
            add:
            - SETPCAP
            - SYS_MODULE
            - SYS_RAWIO
            - SYS_PACCT
            - SYS_ADMIN
            - SYS_NICE
            - SYS_RESOURCE
            - SYS_TIME
            - SYS_TTY_CONFIG
            - MKNOD
            - AUDIT_WRITE
            - AUDIT_CONTROL
            - MAC_OVERRIDE
            - MAC_ADMIN
            - NET_ADMIN
            - SYSLOG
            - CHOWN
            - NET_RAW
            - DAC_OVERRIDE
            - FOWNER
            - DAC_READ_SEARCH
            - FSETID
            - KILL
            - SETGID
            - SETUID
            - LINUX_IMMUTABLE
            - NET_BIND_SERVICE
            - NET_BROADCAST
            - IPC_LOCK
            - IPC_OWNER
            - SYS_CHROOT
            - SYS_PTRACE
            - SYS_BOOT
            - LEASE
            - SETFCAP
            - WAKE_ALARM
            - BLOCK_SUSPEND
          privileged: false
          seLinuxOptions:
            level: s0
            role: system_r
            type: super_t
            user: system_u
          seccompProfile:
            type: Unconfined
        volumeMounts:
        - mountPath: /var/log/
          name: log
          readOnly: true
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /custom-config
          name: custom-config
          readOnly: true
        - mountPath: /dev/kmsg
          name: kmsg
          readOnly: true
        - mountPath: /etc/machine-id
          name: machine-id
          readOnly: true
        - mountPath: /run/systemd/system/
          name: systemd
        - mountPath: /var/run/dbus/
          name: dbus
        - mountPath: /var/run/containerd/containerd.sock
          name: containerd
          readOnly: true
      hostIPC: true
      hostNetwork: true
      hostPID: true
      priorityClassName: system-node-critical
      serviceAccountName: node-problem-detector
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        operator: Exists
      volumes:
      - hostPath:
          path: /var/log/
        name: log
      - hostPath:
          path: /etc/localtime
          type: FileOrCreate
        name: localtime
      - configMap:
          defaultMode: 493
          name: node-problem-detector-custom-config
        name: custom-config
      - hostPath:
          path: /dev/kmsg
        name: kmsg
      - hostPath:
          path: /etc/machine-id
          type: File
        name: machine-id
      - hostPath:
          path: /run/systemd/system/
          type: Directory
        name: systemd
      - hostPath:
          path: /var/run/dbus/
          type: Directory
        name: dbus
      - hostPath:
          path: /var/run/containerd/containerd.sock
          type: Socket
        name: containerd
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

In the container:

strace -s 1024 head -n 1 /dev/kmsg
execve("/usr/bin/head", ["head", "-n", "1", "/dev/kmsg"], 0x7ffe54e2eea8 /* 51 vars */) = 0
brk(NULL)                               = 0x563412481000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=7302, ...}) = 0
mmap(NULL, 7302, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5179297000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@>\2\0\0\0\0\0@\0\0\0\0\0\0\0\340\3\35\0\0\0\0\0\0\0\0\0@\08\0\f\0@\0@\0?\0\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\240\2\0\0\0\0\0\0\240\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\320\6\32\0\0\0\0\0\320\6\32\0\0\0\0\0\320\6\32\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\210\24\2\0\0\0\0\0\210\24\2\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0 \2\0\0\0\0\0\0 \2\0\0\0\0\0\0 \2\0\0\0\0\0\254\236\25\0\0\0\0\0\254\236\25\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\4\0\0\0\0\300\27\0\0\0\0\0\0\300\27\0\0\0\0\0\0\300\27\0\0\0\0\0\324\344\4\0\0\0\0\0\324\344\4\0\0\0\0\0\0\20\0\0\0\0\0\0\1\0\0\0\6\0\0\0h\247\34\0\0\0\0\0h\267\34\0\0\0\0\0h\267\34\0\0\0\0\08P\0\0\0\0\0\0\30\217\0\0\0\0\0\0\0\20\0\0\0\0\0\0\2\0\0\0\6\0\0\0\200\333\34\0\0\0\0\0\200\353\34\0\0\0\0\0\200\353\34\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\340\2\0\0\0\0\0\0\340\2\0\0\0\0\0\0\340\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0h\247\34\0\0\0\0\0h\267\34\0\0\0\0\0h\267\34\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\354\6\32\0\0\0\0\0\354\6\32\0\0\0\0\0\354\6\32\0\0\0\0\0,d\0\0\0\0\0\0,d\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0h\247\34\0\0\0\0\0h\267\34\0\0\0\0\0h\267\34\0\0\0\0\0\2308\0\0\0\0\0\0\2308\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\265\3'[\371\376\345\25\201\375\316\357\227S;\31@5\264\367\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\3\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\363\3\0\0\f\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1", 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1905632, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5179295000
mmap(NULL, 1918592, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f51790c0000
mmap(0x7f51790e2000, 1417216, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f51790e2000
mmap(0x7f517923c000, 323584, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17c000) = 0x7f517923c000
mmap(0x7f517928b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ca000) = 0x7f517928b000
mmap(0x7f5179291000, 13952, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5179291000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f5179296580) = 0
mprotect(0x7f517928b000, 16384, PROT_READ) = 0
mprotect(0x5634118a3000, 4096, PROT_READ) = 0
mprotect(0x7f51792c3000, 4096, PROT_READ) = 0
munmap(0x7f5179297000, 7302)            = 0
brk(NULL)                               = 0x563412481000
brk(0x5634124a2000)                     = 0x5634124a2000
openat(AT_FDCWD, "/dev/kmsg", O_RDONLY) = -1 EPERM (Operation not permitted)
write(2, "head: ", 6head: )                   = 6
write(2, "cannot open '/dev/kmsg' for reading", 35cannot open '/dev/kmsg' for reading) = 35
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++