feat: add containerd-image-verifier-sigstore extension #597

BobyMCbobs · 2025-02-05T21:45:58Z

To enable verifying of every container image run in containerd using Sigstore.

containerd has an ImageVerifier plugin which allows images to be allowed or denied depending on whether they return 0 or 1.

Most of the images delivered through Talos are signed, though problematically some are not.

Using a ClusterImagePolicy (from https://github.com/sigstore/policy-controller), at a system level, things can be defined for allow or deny.

frezbo · 2025-02-05T22:32:39Z

tools/sigstore-policy-tester/README.md

+## Usage
+
+```yaml
+machine:


I would say to put these files as part of extension, so adding this extension enables it. ClusterImagePolicy can be an extension config (though /etc/containers/sigstore needs to be an allowed extension path. possibly create a Talos discussion first)

@frezbo, thank you for taking a look!

I would say to put these files as part of extension, so adding this extension enables it

That's valid, though in Talos' current state, Kubernetes will fail to come up due to several images not being signed and even after that were resolved their workloads won't run without modifying this policy to include their expected keys or OIDC ID/issuer, see:

talosctl images default | xargs -I{} sh -c 'echo {}; cosign verify --certificate-oidc-issuer-regexp='.*' --certificate-identity-regexp='.*' -o text {}' sh

ClusterImagePolicy can be an extension config (though /etc/containers/sigstore needs to be an allowed extension path

as in something like this?

apiVersion: v1alpha1 kind: ExtensionServiceConfig name: sigstore-policy-tester configFiles: - content: | apiVersion: policy.sigstore.dev/v1alpha1 kind: ClusterImagePolicy metadata: name: system spec: images: - glob: "**" authorities: - keyless: url: https://fulcio.sigstore.dev identities: - issuer: https://accounts.google.com subject: k8s-infra-gcr-promoter@k8s-artifacts-prod.iam.gserviceaccount.com ctlog: url: https://rekor.sigstore.dev - keyless: identities: - issuer: https://accounts.google.com subjectRegExp: "@siderolabs\.com$" mountPath: /usr/local/etc/containers/sigstore/policy.yaml # The mount path of the extension service config file. # The environment for the extension service. environment: - CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH=/usr/local/etc/containers/sigstore/policy.yaml

possibly create a Talos discussion first

I'll create one now

siderolabs/talos#10304

BobyMCbobs · 2025-02-06T04:01:20Z

Testing locally building and running with roughly the following

cd ~/github.com/siderolabs/extensions
make docker-sigstore-policy-tester PLATFORM=linux/arm64 TARGET_ARGS="--load"

podman run -it --rm -v "$PWD:$PWD" --workdir "$PWD" \
  ghcr.io/siderolabs/imager:v1.9.3 \
    installer \
    --arch arm64 \
    --system-extension-image ghcr.io/bobymcbobs/siderolabs-extensions/sigstore-policy-tester:latest@sha256:1f0dc82955f4e4861aa13df6fdcd3de6fc5a826a81068148c5d9aa76c146eb36 \
    --output _out

docker image load --input _out/installer-arm64.tar
podman tag ghcr.io/siderolabs/installer:v1.9.3 ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester
podman push ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester

podman run -it --rm -v "$PWD:$PWD" --workdir "$PWD" \
  ghcr.io/siderolabs/imager:v1.9.3 \
    iso \
    --arch arm64 \
    --system-extension-image ghcr.io/bobymcbobs/siderolabs-extensions/sigstore-policy-tester:latest@sha256:1f0dc82955f4e4861aa13df6fdcd3de6fc5a826a81068148c5d9aa76c146eb36 \
    --output _out

Then for the config, using a patch like so

machine:
  env:
    CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH: /var/local/etc/containers/sigstore/policy.yaml
  files:
    - content: |
        [plugins]
          [plugins."io.containerd.image-verifier.v1.bindir"]
            bin_dir = "/usr/local/bin/containerd-image-verifier"
            max_verifiers = 10
            per_verifier_timeout = "10s"
      path: /etc/cri/conf.d/20-customization.part
      op: create
    - content: |
        apiVersion: policy.sigstore.dev/v1alpha1
        kind: ClusterImagePolicy
        metadata:
          name: system
        spec:
          images:
          - glob: "**"
          authorities:
          - keyless:
              url: https://fulcio.sigstore.dev
              identities:
              - issuer: https://accounts.google.com
                subject: k8s-infra-gcr-promoter@k8s-artifacts-prod.iam.gserviceaccount.com
            ctlog:
              url: https://rekor.sigstore.dev
          - keyless:
              identities:
              - issuer: https://accounts.google.com
                subjectRegExp: "@siderolabs\.com$"
      path: /var/local/etc/containers/sigstore/policy.yaml
      op: create

booting up a VM and generating config like so

talosctl gen secrets

talosctl gen config talossigstore https://192.168.64.5:6443 --config-patch @./patch.yaml --install-disk /dev/vda --with-secrets ./secrets.yaml --force --install-image ghcr.io/bobymcbobs/siderolabs/talos/installer:v1.9.3-with-sigstore-policy-tester
talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig apply-config -i --file ./controlplane.yaml

validating the environment with

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls /var/local/etc/containers/sigstore/policy.yaml
NODE           NAME
192.168.64.5   policy.yaml

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls /etc/cri/conf.d/20-customization.part
NODE           NAME
192.168.64.5   20-customization.part

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --depth 2 /usr/local/bin/
NODE           NAME
192.168.64.5   .
192.168.64.5   containerd-image-verifier
192.168.64.5   containerd-image-verifier/containerd-image-verifier-sigstore
192.168.64.5   sigstore-policy-tester

so far, I can't see that there's any affect from the config. Diving on debugging

BobyMCbobs · 2025-02-06T04:24:07Z

logs should appear like the following: https://github.com/containerd/containerd/blob/main/pkg/imageverifier/bindir/bindir.go#L58

I see nothing relevant under the containerd logs

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd

except that the plugin has loaded

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd | grep -i image-verifier
192.168.64.5: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T04:08:03.899137666Z","type":"io.containerd.image-verifier.v1"}

the containerd config looks as expected

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig cat /etc/cri/conf.d/cri.toml
## /etc/cri/conf.d/00-base.part
## /etc/cri/conf.d/01-registries.part
## /etc/cri/conf.d/20-customization.part

version = 3

[debug]
  level = 'debug'

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

      [plugins.'io.containerd.cri.v1.images'.registry.configs]

  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          base_runtime_spec = '/etc/cri/conf.d/base-spec.json'

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/usr/local/bin/containerd-image-verifier'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.nri.v1.nri']
    disable = true

the program is executable

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --long /usr/local/bin/containerd-image-verifier/containerd-image-verifier-sigstore
NODE           MODE         UID   GID   SIZE(B)   LASTMOD           LABEL                                              NAME
192.168.64.5   -rwxr-xr-x   0     0     2650580   Feb  6 16:38:45   system_u:object_r:container_file_t:s0:c325,c705   containerd-image-verifier-sigstore

frezbo · 2025-02-06T07:23:57Z

logs should appear like the following: https://github.com/containerd/containerd/blob/main/pkg/imageverifier/bindir/bindir.go#L58

I see nothing relevant under the containerd logs

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd

except that the plugin has loaded

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd | grep -i image-verifier
192.168.64.5: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T04:08:03.899137666Z","type":"io.containerd.image-verifier.v1"}

the containerd config looks as expected

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig cat /etc/cri/conf.d/cri.toml
## /etc/cri/conf.d/00-base.part
## /etc/cri/conf.d/01-registries.part
## /etc/cri/conf.d/20-customization.part

version = 3

[debug]
  level = 'debug'

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

      [plugins.'io.containerd.cri.v1.images'.registry.configs]

  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          base_runtime_spec = '/etc/cri/conf.d/base-spec.json'

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/usr/local/bin/containerd-image-verifier'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.nri.v1.nri']
    disable = true

the program is executable

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --long /usr/local/bin/containerd-image-verifier/containerd-image-verifier-sigstore
NODE           MODE         UID   GID   SIZE(B)   LASTMOD           LABEL                                              NAME
192.168.64.5   -rwxr-xr-x   0     0     2650580   Feb  6 16:38:45   system_u:object_r:container_file_t:s0:c325,c705   containerd-image-verifier-sigstore

check the cri logs too, containerd is only used to pull etcd and kubelet

frezbo · 2025-02-06T07:24:35Z

machine:
  env:
    CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH: /var/local/etc/containers/sigstore/policy.yaml

I wonder if we can avoid global env's

smira · 2025-02-06T09:41:57Z

I don't quite like the name of the extensions, it should probably be around image-verifier naming, or the fact that it's a plugin.

I'm not personally excited about the imager verifier API in containerd (of course it has nothing to do with this change), but I see the need to have such verification.

smira · 2025-02-06T09:49:19Z

tools/sigstore-policy-tester/README.md

+  files:
+    - content: |
+        [plugins]
+          [plugins."io.containerd.image-verifier.v1.bindir"]


I also would like to check how this plugs in into the image pull process, there are multiple paths in the CRI/Talos itself before we declare this as a production solution.

From the docs (https://github.com/containerd/containerd/blob/main/docs/image-verification.md#image-pull-judgement), it won't pull an image unless it passes the checks in the bindir

Return an exit code of 0 to allow the image to be pulled and any other exit code to block the image from being pulled.

The problem is that containerd doesn't pull images ;)

In fact, they have Transfer API which is only used by ctr, and direct pull, when the actual pull process happens outside of containerd. This API is being used by CRI & Talos. So image verification might be pulled into CRI, but it I doubt it covers all paths.

According to https://youtu.be/fDDSg2NVwO4?t=322, containerd 2.1 will resolve this.

In any case, the functionality of image verification is what I'd like and it doesn't have to be this way. That said, there's work along this path to achieve this functionality

BobyMCbobs · 2025-02-06T19:33:29Z

I don't quite like the name of the extensions, it should probably be around image-verifier naming, or the fact that it's a plugin.

This is fair. What do you think about containerd-imageverifier-sigstore?

BobyMCbobs · 2025-02-06T19:39:19Z

logs should appear like the following: https://github.com/containerd/containerd/blob/main/pkg/imageverifier/bindir/bindir.go#L58
I see nothing relevant under the containerd logs

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd

except that the plugin has loaded

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig logs containerd | grep -i image-verifier
192.168.64.5: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T04:08:03.899137666Z","type":"io.containerd.image-verifier.v1"}

the containerd config looks as expected

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig cat /etc/cri/conf.d/cri.toml
## /etc/cri/conf.d/00-base.part
## /etc/cri/conf.d/01-registries.part
## /etc/cri/conf.d/20-customization.part

version = 3

[debug]
  level = 'debug'

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

      [plugins.'io.containerd.cri.v1.images'.registry.configs]

  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          base_runtime_spec = '/etc/cri/conf.d/base-spec.json'

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/usr/local/bin/containerd-image-verifier'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.nri.v1.nri']
    disable = true

the program is executable

talosctl -e 192.168.64.5 -n 192.168.64.5 --talosconfig ./talosconfig ls --long /usr/local/bin/containerd-image-verifier/containerd-image-verifier-sigstore
NODE           MODE         UID   GID   SIZE(B)   LASTMOD           LABEL                                              NAME
192.168.64.5   -rwxr-xr-x   0     0     2650580   Feb  6 16:38:45   system_u:object_r:container_file_t:s0:c325,c705   containerd-image-verifier-sigstore

check the cri logs too, containerd is only used to pull etcd and kubelet

The only logs for cri that are relevant are

talosctl -e 192.168.64.6 -n 192.168.64.6 --talosconfig ./talosconfig logs cri | grep verif
192.168.64.6: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2025-02-06T19:36:11.001063426Z","type":"io.containerd.image-verifier.v1"}

machine:
  env:
    CONTAINERD_IMAGE_VERIFIER_SIGSTORE_POLICY_PATH: /var/local/etc/containers/sigstore/policy.yaml

I wonder if we can avoid global env's

The value can be overridden via an ldflag on build. I'll update it.

To enable verifying of every container image run through containerd

BobyMCbobs force-pushed the add-sigstore-policy-tester branch from fbd7fd9 to ea1c078 Compare February 5, 2025 21:53

frezbo reviewed Feb 5, 2025

View reviewed changes

BobyMCbobs force-pushed the add-sigstore-policy-tester branch from ea1c078 to 0a6e49f Compare February 6, 2025 02:34

smira reviewed Feb 6, 2025

View reviewed changes

BobyMCbobs force-pushed the add-sigstore-policy-tester branch 3 times, most recently from 7df0880 to a0ef75e Compare February 6, 2025 19:54

feat: add sigstore-policy-tester extension

3066da4

To enable verifying of every container image run through containerd

BobyMCbobs force-pushed the add-sigstore-policy-tester branch from a0ef75e to 3066da4 Compare February 6, 2025 21:02

BobyMCbobs changed the title ~~feat: add sigstore-policy-tester extension~~ feat: add containerd-image-verifier-sigstore extension Feb 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add containerd-image-verifier-sigstore extension #597

feat: add containerd-image-verifier-sigstore extension #597

BobyMCbobs commented Feb 5, 2025 •

edited

Loading

frezbo Feb 5, 2025

BobyMCbobs Feb 6, 2025 •

edited

Loading

BobyMCbobs Feb 6, 2025

BobyMCbobs commented Feb 6, 2025 •

edited

Loading

BobyMCbobs commented Feb 6, 2025 •

edited

Loading

frezbo commented Feb 6, 2025

frezbo commented Feb 6, 2025

smira commented Feb 6, 2025

smira Feb 6, 2025

BobyMCbobs Feb 6, 2025

smira Feb 7, 2025

BobyMCbobs Feb 8, 2025

BobyMCbobs commented Feb 6, 2025

BobyMCbobs commented Feb 6, 2025

feat: add containerd-image-verifier-sigstore extension #597

Are you sure you want to change the base?

feat: add containerd-image-verifier-sigstore extension #597

Conversation

BobyMCbobs commented Feb 5, 2025 • edited Loading

frezbo Feb 5, 2025

Choose a reason for hiding this comment

BobyMCbobs Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

BobyMCbobs Feb 6, 2025

Choose a reason for hiding this comment

BobyMCbobs commented Feb 6, 2025 • edited Loading

BobyMCbobs commented Feb 6, 2025 • edited Loading

frezbo commented Feb 6, 2025

frezbo commented Feb 6, 2025

smira commented Feb 6, 2025

smira Feb 6, 2025

Choose a reason for hiding this comment

BobyMCbobs Feb 6, 2025

Choose a reason for hiding this comment

smira Feb 7, 2025

Choose a reason for hiding this comment

BobyMCbobs Feb 8, 2025

Choose a reason for hiding this comment

BobyMCbobs commented Feb 6, 2025

BobyMCbobs commented Feb 6, 2025

BobyMCbobs commented Feb 5, 2025 •

edited

Loading

BobyMCbobs Feb 6, 2025 •

edited

Loading

BobyMCbobs commented Feb 6, 2025 •

edited

Loading

BobyMCbobs commented Feb 6, 2025 •

edited

Loading