certificates-relation-changed hook failed: creating secrets: Regular expression is invalid: nothing to repeat

[rpbritton]

Bug Description

Solutions QA as seen a run when deploying cos on microk8s 1.28/stable where the catalogue unit stays in error.

To Reproduce

deploy cos-lite bundle on top of MicroK8s 1.28 with Juju 3.5.

Environment

baremetal:
  distro_series: focal
cos:
  cos-lite: latest/stable:11
  distro_series: kubernetes
cos-workload-test:
  distro_series: kubernetes
existing_juju_maas_controller_kubernetes:
  juju: 3.5.1
existing_juju_maas_controller_microk8s:
  juju: 3.5.1
fce_container_image:
  fce-container-image: ubuntu:jammy
juju_kubernetes_controller:
  juju: 3.5.1
juju_maas_controller:
  juju: 3.5.1
juju_microk8s_controller:
  juju: 3.5.1
kubernetes-maas:
  ceph_channel: quincy/stable
  charmed-kubernetes: '1.29'
  cloud-init: 24.1.3-0ubuntu1~22.04.5
  distro_series: jammy
local:
  cpe-foundation: 2.21.2+git.57.g09b536b5
  distro_series: !!binary |
    amFtbXk=
  sku: fkb-master-kubernetes-jammy-baremetal-flannel
maas:
  maas: 3.4.2-
metallb_kubernetes:
  distro_series: kubernetes
metallb_microk8s:
  distro_series: kubernetes
microk8s:
  cloud-init: 24.1.3-0ubuntu1~22.04.5
  distro_series: jammy
  microk8s: v1.29.0
==================================
charm-catalogue-k8s-latest/stable:
  channel: latest/stable
  last-updated: 2024-06-21 00:00:00
  name: catalogue-k8s
  package-type: charm
  publisher: Canonical Observability
  revision: 39
  series: focal
  version: null

Relevant log output

The units debug logs in the crashdump show the following being repeated:

unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.kubernetes.provider opening model "cos".
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for cps7f4vmp25c74prh1pg-1 of /v1, Kind=Secret in namespace cos
unit-catalogue-0: 2024-06-23 19:29:24 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: Regular expression is invalid: nothing to repeat
secret with label "ca-certificate-chain" already exists
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of /v1, Kind=ServiceAccount in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of /v1, Kind=ServiceAccount in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.kubernetes.provider opening model "cos".
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for cps7f4vmp25c74prh1p0-1 of /v1, Kind=Secret in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of /v1, Kind=ServiceAccount in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of /v1, Kind=ServiceAccount in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-e963a74d-1f92-4530-8f28-f1997b525969: 2024-06-23 19:29:24 DEBUG juju.worker.caasadmission received admission request for unit-catalogue-0 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.kubernetes.provider opening model "cos".
unit-catalogue-0: 2024-06-23 19:29:24 ERROR juju.worker.uniter.operation hook "certificates-relation-changed" (via hook dispatching script: dispatch) failed: creating secrets: Regular expression is invalid: nothing to repeat
secret with label "ca-certificate-chain" already exists
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.machinelock machine lock "machine-lock" released for catalogue/0 uniter (run relation-changed (33; app: ca) hook)
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.worker.uniter.operation lock released for catalogue/0
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.worker.uniter.actions no next action from pending=[]; completed=map[]
unit-catalogue-0: 2024-06-23 19:29:24 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-catalogue-0: 2024-06-23 19:29:24 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "certificates-relation-changed"

Additional context

No response

[lucabello]

We should look into CertHandler and see if we're trying to create the ca-certificate-chain secret without checking it already exists, or dig deeper in the tls_certificates library.