Sidecar plugin - plugin sidecar failed. open /home/argocd/cmp-server/plugins: no such file or directory

[cyrus-mc]

Trying to upgrade from 2.2.5 to latest (2.5.2) and convert my custom plugins to the new sidecar approach.

To familiarize myself with the process I am attempting to implement the plugin in the documentation.

I have the following configMap:

apiVersion: v1
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: cmp-plugin
    spec:
      version: v1.0
      init:
        command: [sh, -c, 'echo "Initializing..."']
      generate:
        command: [sh, -c, 'echo "{\"kind\": \"ConfigMap\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"$ARGOCD_APP_NAME\", \"namespace\": \"$ARGOCD_APP_NAMESPACE\", \"annotations\": {\"Foo\": \"$FOO\", \"KubeVersion\": \"$KUBE_VERSION\", \"KubeApiVersion\": \"$KUBE_API_VERSIONS\",\"Bar\": \"baz\"}}}"']
      discover:
        find:
          command: [sh, -c, find . -name env.yaml]
kind: ConfigMap

I then add a sidecar container to the repo-server deployment as such:

      containers:
      - args:
        - --loglevel
        - debug
        command:
        - /var/run/argocd/argocd-cmp-server
        image: busybox
        imagePullPolicy: Always
        name: cmp
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          name: cmp-plugin
          subPath: plugin.yaml
        - mountPath: /tmp
          name: cmp-tmp

When repo-server starts the CMP container is running and if I exec into it I see

• /home/argocd/cmp-server/config/plugin.yaml - with expected contents
• /home/argocd/cmp-server/plugins/cmp-plugin-v1.0.sock

My application object is as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    app.kubernetes.io/instance: main
  name: node-canary-test
  namespace: argocd
spec:
  destination:
    namespace: app-test
    server: https://kubernetes.default.svc
  project: main
  source:
    path: node-canary
    plugin: {}
    repoURL: https://github.com/xxxxx/gitops.git
    targetRevision: HEAD
  syncPolicy:
    syncOptions:
    - CreateNamespace=true

Within repo gitops, path node-canary exist a file env.yaml required to satisfy the discover command specified in plugin.yaml.

Yet when Argo tries to render manifests it errors with:

rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. open /home/argocd/cmp-server/plugins: no such file or directory

I even enabled debug log for argocd-cmp-server yet it provides no additional information in the cmp container.

› kubectl logs -f argocd-repo-server-7bf5c95b6c-8f25k cmp
time="2022-11-27T04:55:38Z" level=info msg="ArgoCD ConfigManagementPlugin Server is starting" built="2022-11-07T16:42:47Z" commit=148d8da7a996f6c9f4d102fdd8e688c2ff3fd8c7 version=v2.5.2+148d8da
time="2022-11-27T04:55:38Z" level=info msg="argocd-cmp-server v2.5.2+148d8da serving on /home/argocd/cmp-server/plugins/cmp-plugin-v1.0.sock"
time="2022-11-27T05:05:38Z" level=info msg="Alloc=7585 TotalAlloc=19792 Sys=29265 NumGC=7 Goroutines=7"

Repo server logs aren't very helpful either

time="2022-11-27T04:57:43Z" level=info msg="manifest error cache miss: &ApplicationSource{RepoURL:https://github.com/xxxxxxxx/gitops.git,Path:node-canary,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:,Env:[]*EnvEntry{&EnvEntry{Name:APPLICATION,Value:node-canary,},&EnvEntry{Name:ENVIRONMENT,Value:non-prod,},&EnvEntry{Name:DEPLOYMENT,Value:test,},&EnvEntry{Name:REGION,Value:us-west-2,},&EnvEntry{Name:AWS_ACCOUNT_ID,Value:xxxxxx,},&EnvEntry{Name:DOMAIN,Value:infrastructure,},},},Chart:,}/04eb4a95afc0fdfb0dd72c82b826558835eeb241"
time="2022-11-27T04:57:43Z" level=error msg="finished unary call with code Unknown" error="plugin sidecar failed. open /home/argocd/cmp-server/plugins: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2022-11-27T04:57:42Z" grpc.time_ms=1360.826 span.kind=server system=grpc

[cyrus-mc]

It is almost like it is trying to run the plugin locally in the argocd server container and not the sidecar.

Is there a way to verify what plugins are registered with ArgoCD? The old config method it was easy to see via configuration what was defined. The sidecar, outside of it running it doesn't really provide any information on whether it registered successfully with Argo.

Nor does the logs provide any information on what sidecar plugin it is executing in (if you had multiple sidecar plugins each with different discovery rules).

[cyrus-mc]

Reading - https://github.com/crenshaw-dev/argo-cd/blob/d8642fd95108673d7dcdbd9147eb5d9d41471150/docs/proposals/config-management-plugin-v2.md

I discovered my issue. My repo-server deployment was incorrect as it was missing the mount point for plugins-home.