Skip to content

Latest commit

 

History

History
212 lines (159 loc) · 11 KB

ROTATION.md

File metadata and controls

212 lines (159 loc) · 11 KB

Secrets Provider - Secrets Rotation

Table of Contents

Overview

The secrets rotation feature detailed below allows Kubernetes applications to refresh Conjur secrets if there are any changes to the secrets.

How Secrets Rotation Works

how secrets rotation works

Note: see how-push-to-file-works for more detail on how Push to File works.

  1. The Secrets Provider authenticates to the Conjur server using the Kubernetes Authenticator (conjur-authn-k8s-client).

  2. The Secrets Provider reads all Conjur secrets required across all secret groups.

  3. The Secrets Provider sidecar container starts up and retrieves the initial secrets. If secrets rotation is enabled, after the duration specified by conjur.org/secrets-refresh-interval or the default interval the Secrets Provider will check if the secrets have changed by comparing the SHA-256 checksums of the secrets with the previous checksums. The Secrets Provider does not save any of the unencrypted secrets. If the time needed to fetch the secrets is longer than is specified for the duration, then the duration will be the actual time to retrieve the secrets.

    For example: If the duration is set to two seconds, but retrieving the secrets takes three second then the secrets will be updated every three seconds.

    Note: If one or more of the secrets have been removed from Conjur or have had access revoked, the Secrets Provider will remove the secrets files from the shared volume. To disable this feature, set the conjur.org/remove-deleted-secrets-enabled annotation to false.

  4. The Secrets Provider renders secret files for each secret group, and writes the resulting files to a volume that is shared with your application container. The Secrets Provider will rewrite the secret files if there are any changes.

  5. The application reads the secrets.

  6. The application can optionally delete the secret files after consuming. If the secret files are deleted, they will only be recreated when the secret values have changed.

Set up Secrets Provider for secrets rotation

There are two new annotations introduced and one annotation is updated.

Prerequisites:

Requires secrets-provider-for-k8s v1.4.0 or later.

For Push to File mode

Follow the procedure to set up Secrets Provider for Push to File

For Kubernetes Secrets mode

Follow the procedure to set up Kubernetes Secrets

Modify the Kubernetes manifest

  1. Change the Secrets provider container to be a sidecar. If it was configured as an init container remove the initContainers so the image is in the containers section as below:

     spec:
   containers:
   - image: secrets-provider-for-k8s:latest

  2. Update the conjur.org/container-mode annotation:

    conjur.org/container-mode: sidecar

  3. Add the new Secrets Rotation annotations. There are two new annotations added, only one of the annotations is required to be set to enable secrets rotation.

    conjur.org/secrets-refresh-enabled enables the feature if the container mode is sidecar. The default duration is 5 minutes if the duration is not specified with the conjur.org/secrets-refresh-interval.

    conjur.org/secrets-refresh-interval Sets the duration and is a string as defined here. Setting a time implicitly enables refresh. Valid time units are s, m, and h (for seconds, minutes, and hours, respectively). Some examples of valid duration strings:

    • 5m
    • 2h30m
    • 48h
    The minimum refresh interval is 1 second. A refresh interval of 0 seconds is treated as a fatal configuration error.

    conjur.org/secrets-refresh-enabled: "true"
conjur.org/secrets-refresh-interval: 10m

    Here is the example push file manifest modified for rotation.

    For comparison, a push to file manifest as an init container.

    Secrets Rotation can also be used with Kubernetes secrets.

    Here is an example Kubernetes Secrets manifest modified for rotation.

    For comparison, a Kubernetes Secrets manifest as an init container.

    Secrets Rotation can also be used with JWT authentication. Here is an example JWT based push to file manifest modified for rotation.

    Here is an example JWT based Kubernetes secrets manifest modified for rotation.

Using sentinel files for checking provider status

Prerequisites for using sentinel files:

Requires secrets-provider-for-k8s v1.4.1 or later.
Requires a shared volume that both the application and Secrets Provider can read and write to.

Secrets Provider allows for its status to be monitored through the creation of a couple of empty sentinel files: CONJUR_SECRETS_PROVIDED and CONJUR_SECRETS_UPDATED. The first file is created when the SP has completed its first round of providing secrets via secret files / Kubernetes Secrets. It creates/recreates the second file whenever it has updated secret files / Kubernetes Secrets. If desirable, application containers can mount these files via a shared volume.

The Pod would need a Volume defined:

    volumes:
    - name: conjur-status
      emptyDir:
        medium: Memory

The application container and SP container would need to include volumeMounts similar to this:

    volumeMounts:
    - mountPath: /conjur/status
      name: conjur-status

These sentinel files can be used to delay the start of the application until after the Secrets provider has started up and written the secrets. Kubelet will start the pod containers in the order they are listed in the manifest. A postStart lifecycle hook can be added to the Secrets Provider manifest which will delay the start of the app container until the postStart lifecycle hook is complete. See conjur-secrets-provided for an example of the Secrets Provider script.

        lifecycle:
          postStart:
            exec:
              command:
              - /usr/local/bin/conjur-secrets-provided.sh

A livenessProbe for an application container that would serve as a "file watcher" can potentially look something like this (assuming the livenessProbe is not already being used by the container as a health probe). This will cause the application to be restarted after there secrets have been updated.

 livenessProbe:
          exec:
            command:
            - /mounted/status/conjur-secrets-unchanged.sh
          failureThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1

By default, the Secrets Provider container runs using a default username secrets-provider, user ID 777, and group ID 777. For the application to delete the sentinel files the app and the Secrets provider should run as the same UID. For example the below securityContext can be added to both the Secrets Provider and the Application, replacing 9999 with your desired value.

        securityContext:
          runAsUser: 9999

Reference Table of Configuration Annotations

In addition to the basic Secrets Provider configuration, below is a list of annotations that are needed for secrets rotation.

K8s Annotation Description
conjur.org/container-mode Configurable values:
  • init
  • application
  • sidecar
Defaults to init.
Must be set to sidecarfor secrets rotation.
conjur.org/secrets-refresh-enabled Set to true to enable Secrets Rotation. Defaults to false unless conjur.org/secrets-refresh-interval is explicitly set. Secrets Provider will exit with error if this is set to false and conjur.org/secrets-refresh-interval is set.
conjur.org/secrets-refresh-interval Set to a valid duration string as defined here. Setting a time implicitly enables refresh. Valid time units are s, m, and h (for seconds, minutes, and hours, respectively). Some examples of valid duration strings:
  • 5m
  • 2h30m
  • 48h
The minimum refresh interval is 1 second. A refresh interval of 0 seconds is treated as a fatal configuration error. The default refresh interval is 5 minutes. The maximum refresh interval is approximately 290 years.
conjur.org/remove-deleted-secrets-enabled Set to false to disable deletion of secrets files from the shared volume when a secret is removed or access is revoked in Conjur. Defaults to true.

Troubleshooting

This section describes how to troubleshoot common Secrets Provider for Kubernetes issues.

To enable the debug logs, See enable-logs

Issue Error code Resolution
No change in secret files, no secret files written CSPFK018I This is an info message and not an error. It indicates that the Secrets Provider did not detect a change in secrets for a secrets group. The secret file for this group will not be written. Note: there may be changes in other secret groups and those files will be written.
Invalid secrets refresh interval annotation CSPFK050E There is an error with the interval annotation, check the log message for the exact failure reason. See the annotation reference for more information on setting the annotations.
Invalid secrets refresh configuration CSPFK051E Secrets refresh is enabled either by setting conjur.org/secrets-refresh-enabled to true or setting a duration for conjur.org/secrets-refresh-interval and the mode is not sidecar. The mode must be sidecar.