failed to wait for configauditreport caches to sync

[malvery]

What steps did you take and what happened:
trivy-operator periodical gets CrashLoopBackOff on configauditreport or rbacauditreport caches sync after 30s. I sets controllerCacheSyncTimeout: "10m", but it does't help, it stills 30s in the log.

unable to run trivy operator: starting controllers manager: [failed to wait for configauditreport caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.ConfigAuditReport, failed waiting for all runnables to end within grace period of 30s: context deadline exceeded]

What did you expect to happen:

Anything else you would like to add:
trivy-operator-config:

data:
  CONTROLLER_CACHE_SYNC_TIMEOUT: 10m
  OPERATOR_ACCESS_GLOBAL_SECRETS_SERVICE_ACCOUNTS: "true"
  OPERATOR_BATCH_DELETE_DELAY: 10s
  OPERATOR_BATCH_DELETE_LIMIT: "10"
  OPERATOR_BUILT_IN_TRIVY_SERVER: "false"
  OPERATOR_CACHE_REPORT_TTL: 120h
  OPERATOR_CLUSTER_COMPLIANCE_ENABLED: "true"
  OPERATOR_CLUSTER_SBOM_CACHE_ENABLED: "false"
  OPERATOR_CONCURRENT_NODE_COLLECTOR_LIMIT: "1"
  OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT: "10"
  OPERATOR_CONFIG_AUDIT_SCANNER_ENABLED: "true"
  OPERATOR_CONFIG_AUDIT_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: "true"
  OPERATOR_EXPOSED_SECRET_SCANNER_ENABLED: "true"
  OPERATOR_HEALTH_PROBE_BIND_ADDRESS: :9090
  OPERATOR_INFRA_ASSESSMENT_SCANNER_ENABLED: "true"
  OPERATOR_LOG_DEV_MODE: "false"
  OPERATOR_MERGE_RBAC_FINDING_WITH_CONFIG_AUDIT: "false"
  OPERATOR_METRICS_BIND_ADDRESS: :8080
  OPERATOR_METRICS_CLUSTER_COMPLIANCE_INFO_ENABLED: "false"
  OPERATOR_METRICS_CONFIG_AUDIT_INFO_ENABLED: "false"
  OPERATOR_METRICS_EXPOSED_SECRET_INFO_ENABLED: "false"
  OPERATOR_METRICS_FINDINGS_ENABLED: "true"
  OPERATOR_METRICS_IMAGE_INFO_ENABLED: "false"
  OPERATOR_METRICS_INFRA_ASSESSMENT_INFO_ENABLED: "false"
  OPERATOR_METRICS_RBAC_ASSESSMENT_INFO_ENABLED: "false"
  OPERATOR_METRICS_VULN_ID_ENABLED: "false"
  OPERATOR_PRIVATE_REGISTRY_SCAN_SECRETS_NAMES: '{}'
  OPERATOR_RBAC_ASSESSMENT_SCANNER_ENABLED: "false"
  OPERATOR_SBOM_GENERATION_ENABLED: "true"
  OPERATOR_SCAN_JOB_RETRY_AFTER: 30s
  OPERATOR_SCAN_JOB_TIMEOUT: 5m
  OPERATOR_SCAN_JOB_TTL: 24h
  OPERATOR_SCANNER_REPORT_TTL: 24h
  OPERATOR_SEND_DELETED_REPORTS: "false"
  OPERATOR_VULNERABILITY_SCANNER_ENABLED: "true"
  OPERATOR_VULNERABILITY_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: "true"
  OPERATOR_WEBHOOK_BROADCAST_CUSTOM_HEADERS: ""
  OPERATOR_WEBHOOK_BROADCAST_TIMEOUT: 30s
  OPERATOR_WEBHOOK_BROADCAST_URL: ""
  TRIVY_SERVER_HEALTH_CHECK_CACHE_EXPIRATION: 10h

[Miscellaneous information that will assist in solving the issue.]

Environment:

• Trivy-Operator version (use trivy-operator version): 0.22.0
• Kubernetes version (use kubectl version): v1.29.8-eks-a737599
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 24.04

[afdesk]

@malvery thanks for the report
could you retest it with the latest version of trivy-operator and confirm that the issue still happens?
thanks

[lefterisALEX]

facing similar issue because of the sbomreports. Reason is that we have over time generated quite an ammout of sbomreports since there is no cleanup mechanism afaik.

For example for this deployment we have 6 sbomreports insteadm of having only the latest.

 k get sbomreports -l trivy-operator.container.name=bucketweb
NAME                                               REPOSITORY       TAG                   SCANNER   AGE
replicaset-thanos-bucketweb-569b6bb7d4-bucketweb   bitnami/thanos   0.37.0-debian-12-r0   Trivy     32d
replicaset-thanos-bucketweb-659c89d696-bucketweb   bitnami/thanos   0.36.1-debian-12-r3   Trivy     105d
replicaset-thanos-bucketweb-6df4649458-bucketweb   bitnami/thanos   0.36.1-debian-12-r0   Trivy     119d
replicaset-thanos-bucketweb-76c5768469-bucketweb   bitnami/thanos   0.36.1-debian-12-r3   Trivy     101d
replicaset-thanos-bucketweb-7c95955dbb-bucketweb   bitnami/thanos   0.36.1-debian-12-r3   Trivy     90d
replicaset-thanos-bucketweb-d96686945-bucketweb    bitnami/thanos   0.36.1-debian-12-r2   Trivy     119d

when new trivy-operator pods try to start are crashing due to throttling while they try to read all the sbomreports.