0.22.0 - unable to initialize a remote image scanner

[ChanduReddy123]

What steps did you take and what happened:
we've recently installed the trivy operator (v0.22.0) in our AWS EKS cluster (v1.30.7)
with the following values.yaml

trivy:
  ignoreUnfixed: true
  server:
    securityContext:
      allowPrivilegeEscalation: false
excludeNamespaces: "kube-system"
operator:
  builtInTrivyServer: true

trivyOperator:
  scanJobPodTemplatePodSecurityContext:
    RunAsUser: 10000
    RunAsGroup: 10000
    RunAsNonRoot: true

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"
  prometheus.io/scrape_interval: "5m"

podSecurityContext:
  runAsUser: 65534
  runAsNonRoot: true
  fsGroup: 65534

nodeCollector:
  useNodeSelector: false

used the following command to install the trivy operator

helm install trivy-operator aqua/trivy-operator \
                                        --namespace trivy-system \
                                        --create-namespace \
                                        --version 0.24.0 \
                                        --values values.yaml

we keep getting errors in the trivy-operator pod and few scan job pods exit with error

The error in the trivy-operator is as follows
{"level":"error","ts":"2025-01-01T08:49:56Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-75ff45f945","container":"istio-proxy","status.reason":"Error","status.message":"2025-01-01T08:49:53Z\tFATAL\tFatal error\timage scan error: scan error: unable to initialize a scanner: unable to initialize a remote image scanner: unable to find the specified image \"auto\" in [\"docker\" \"containerd\" \"podman\" \"remote\"]: 4 errors occurred:\n\t* docker error: unable to inspect the image (auto): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* containerd error: containerd socket not found: /run/containerd/containerd.sock\n\t* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* remote error: GET https://index.docker.io/v2/library/auto/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/auto Type:repository]]\n\n\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

we also tried to run the scan pods with root user but we still got the similar error.

we were also getting the issues with kube-system namespace pods so for time being we've excluded the namespace from the scan

What did you expect to happen:
no errors recorded in the trivy-operator pod and vulnerability report to be generated for all the images running in the cluster

Environment:

• Trivy-Operator version (v0.22.0):
• Kubernetes version (v1.30.7):
• OS (Linux amd64 Amazon Linux 2023.6.20241121):

[tom1299]

This error seems to be related to missing permissions to pull the image auto from dockerhub:

remote error: GET https://index.docker.io/v2/library/auto/manifests/latest: UNAUTHORIZED: authentication required;

If get a similar error when trying to pull the image directly:

$ podman pull docker.io/library/auto:latest
Trying to pull docker.io/library/auto:latest...
Error: initializing source docker://auto:latest: reading manifest latest in docker.io/library/auto: requested access to the resource is denied

[ChanduReddy123]

we never used any image auto in our deployments
auto image is related to the istio
I don't think it is private image

[tom1299]

we never used any image auto in our deployments auto image is related to the istio I don't think it is private image

You are right, auto is just a placeholder. See here. So trivy will neither be able to pull or scan the image. It seems like the problem is more related to the istio configuration.

[ChanduReddy123]

what I obseved is trivy operator is not able to pull the system pod images like

• .dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.6-eksbuild.1
• .dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.19.2-eksbuild.1

if these pods are from private registries how did EKS pull them into the cluster
if it used some creds to pull the images why can't trivy-operator use those creds to pull the images