Skip to content

Latest commit

 

History

History
402 lines (297 loc) · 30.8 KB

README.md

File metadata and controls

402 lines (297 loc) · 30.8 KB

Steward Helm Chart

Install and configure Steward on Kubernetes.

Table of Content

Prerequisites

This Helm chart requires Helm 3 or higher.

The Steward Helm chart is currently not published in any public Helm repository. Therefore it must be installed from a source checkout.

Install

Use the helm install command to install the Steward Helm chart:

helm install RELEASE_NAME CHECKOUT_DIR/charts/steward/ ...

The helm install command has a parameter --namespace that defines the target namespace of the release. Normally this is the namespace where the application will be installed to. Helm also stores the release data in that namespace. However, the Steward chart does not use the release namespace but has a separate parameter targetNamespace.name defining the namespace where Steward will be installed to. This allows to include the Steward chart as dependency into another chart but still install into an own namespace. The Helm release target namespace and the Steward target namespace can be equal if required.

If the Steward target namespace deliberately exists already, parameter targetNamespace.create should be set to false to suppress a resource conflict error.

Do not use the --no-hooks option of the helm install command. Hooks are required for a consistent installation.

Upgrade

Use the helm upgrade command to upgrade Steward releases:

helm upgrade RELEASE_NAME CHECKOUT_DIR/charts/steward/ ...

To reuse values from the current release revision, do NOT use the --reuse-values option of the helm upgrade command. This option will not only reuse overridden values, but also the built-in values of the current release's chart version. The result might be unexpected. Instead:

  1. Retrieve only the overridden values from the current release:

    helm get values RELEASE_NAME --namespace RELEASE_NAMESPACE --output yaml \
    >prev-values.yaml

  2. Apply the overridden values to the upgrade, optionally adding more overrides:

    helm upgrade ... -f prev-values.yaml -f new-values.yaml --set ...

    Note the order of increasing precedence from left to right!

Uninstall

Use the helm uninstall command to delete a Steward release:

helm uninstall RELEASE_NAME ...

Note that Steward's custom resource definitions will not be deleted automatically (see Custom Resource Definitions below).

Chart Configuration

The tables in the following sections list the configurable parameters of the Steward chart.

Target Namespace

Parameter Description Default
targetNamespace.create
bool		 Whether to create the target namespace. Can be set to false if the namespace exists already, e.g. because the target namespace is also the target namespace of the Helm release and therefore must be created before installing the Chart. true
targetNamespace.name
string		 The name of the namespace where Steward should be installed to. Note that we do not use the Helm release target namespace, so that this chart can be used as subchart of another chart and still installs into its dedicated namespace. steward-system

Pipeline Run Controller

Parameter Description Default
runController.image.repository
string		 The container registry and repository of the Run Controller image. stewardci/stewardci-run-controller
runController.image.tag
string		 The tag of the Run Controller image in the container registry. A fixed image tag.
runController.image.pullPolicy
string		 The image pull policy for the Run Controller image. For possible values see field imagePullPolicy of the container spec in the Kubernetes API documentation. IfNotPresent
runController.resources
object of RecourceRequirements		 The resource requirements of the Run Controller container. If not set or empty, the default value is used. {"limits": {"cpu":1, "memory":"256Mi"}, "requests": {"cpu":"100m", "memory":"256Mi"}}
runController.podSecurityContext
object of PodSecurityContext		 The pod security context of the Run Controller pod. If not set or empty, the default value is used. {}
runController.securityContext
object of SecurityContext		 The security context of the Run Controller container. If not set or empty, the default value is used. {"privileged":false, "seccompProfile": {"type":"RuntimeDefault"}, "allowPrivilegeEscalation":false, "capabilities": {"drop": ["ALL"]}, "readOnlyRootFilesystem":true, "runAsNonRoot":true, "runAsUser":1000, "runAsGroup":"1000}"}
runController.nodeSelector
object		 The nodeSelector field of the Run Controller pod spec. If not set or empty, the default value is used. {}
runController.affinity
object of Affinity		 The affinity field of the Run Controller pod spec. If not set or empty, the default value is used. {}
runController.tolerations
array of Toleration		 The tolerations field of the Run Controller pod spec. If not set or empty, the default value is used. []
runController.args.qps
integer		 The maximum queries per second (QPS) from the controller to the cluster. 5
runController.args.burst
integer		 The burst limit for throttle connections (maximum number of concurrent requests). 10
runController.args.threadiness
integer		 The maximum number of reconciliations performed in parallel. 2
runController.args.logVerbosity
integer		 The log verbosity. Levels are adopted from Kubernetes logging conventions. 3
runController.args.heartbeatInterval
duration		 The interval of controller heartbeats. 1m
runController.args.heartbeatLogging
bool		 Whether controller heartbeats should be logged. true
runController.args.heartbeatLogLevel
bool		 The log level to be used for controller heartbeats. 3
runController.args.k8sAPIRequestTimeout
duration		 The timeout for Kubernetes API requests. A value of zero means no timeout. If empty, a default timeout will be applied. empty
runController.podSecurityPolicyName
string		 The name of an existing pod security policy that should be used by the run controller. If empty, a default pod security policy will be created. empty
runController.logging.customLoggingDetails
list		 Define a list of log detail providers. See example below. {}

Custom Logging Details

The custom logging details can be defined at runController.logging.customLoggingDetails. The content is a list, where each entry defines a log detail provider. A log provider definition consists of a map with keys: logKey, kind and spec. The logKey is used as label for the structured logging. The kind defines the used provider. The spec depends on the used kind. The available providers are described below:

Annotation provider

The annotation provider is used to access annotations in the metadata section of a pipeline run. For this kind: annotation is used. In the spec a key to access the annotation must be defined.

runController:
  logging:
    customLoggingDetails:
    - logKey: customLogKey1
      kind: annotation
      spec:
        key: foo.bar.annotationKey1

With a pipelineRuns metadata:

metadata:
  annotations:
    foo.bar.annotationKey1: annotationValue

This would result in the following additional log entry for a pipeline run related log:

customLogKey1: annotationValue
Label provider

The label provider is used to access labels in the metadata section of a pipeline run. For this kind: label is used. In the spec a key to access the label must be defined.

runController:
  logging:
    customLoggingDetails:
    - logKey: customLogKey1
      kind: label
      spec:
        key: foo.bar.labelKey1

With a pipelineRuns metadata:

metadata:
  labels:
    foo.bar.labelKey1: labelValue

This would result in the following additional log entries for a pipeline run related log:

customLogKey2: labelValue

Common parameters:

Parameter Description Default
imagePullSecrets
array of LocalObjectReference		 The image pull secrets to be used for pulling controller images. []

Monitoring

Parameter Description Default
metrics.serviceMonitors.enabled
bool		 Whether to generate ServiceMonitor resource for Prometheus Operator. false
metrics.serviceMonitors.extraLabels
object of string		 Labels to be attached to the ServiceMonitor resources for Prometheus Operator. {}

Pipeline Runs

Parameter Description Default
pipelineRuns.podSecurityPolicyName
string		 The name of an existing pod security policy that should be used by pipeline run pods. If empty, a default pod security policy will be created.
pipelineRuns.timeout
duration		 The maximum execution time of pipelines. 60m
pipelineRuns.networkPolicy
string		 Deprecated: Use pipelineRuns.networkPolicies instead.
pipelineRuns.defaultNetworkPolicyName The name of the network policy which is used when no network profile is selected by a pipeline run spec. default if pipelineRuns.networkPolicies is not set or empty.
pipelineRuns.networkPolicies
map[string]string		 The network policies selectable as network profiles in pipeline run specs. The key can be any valid YAML key not starting with underscore (_). The value must be a string containing a complete networkpolicy.networking.k8s.io resource manifest in YAML format. The .metadata section of the manifest can be omitted, as it will be replaced anyway. See the Kubernetes documentation of network policies for details about Kubernetes network policies.

Note that Steward ensures that all pods in pipeline run namespaces are isolated in terms of network policies. The policy defined here adds egress and/or ingress rules.		 A single entry named default whose value is a network policy defining rules that allow ingress traffic from all pods in the same namespace and egress traffic to the internet, the cluster DNS resolver.
pipelineRuns.limitRange
string		 The limit range to be created in every pipeline run namespace. The value must be a string containing a complete limitrange resource manifest in YAML format. The .metadata section of the manifest can be omitted, as it will be replaced anyway. See the Kubernetes documentation of limit ranges for details about Kubernetes limit ranges. A limit range defining a default CPU request of 0.5 CPUs, a default CPU limit of 3 CPUs, a default memory request of 0.5 GiB and a default memory limit of 3 GiB.

This default limit range might change with newer releases of Steward. It is recommended to set an own limit range to avoid unexpected changes with Steward upgrades.
pipelineRuns.resourceQuota
string		 The resource quota to be created in every pipeline run namespace. The value must be a string containing a complete resourcequotas resource manifest in YAML format. The .metadata section of the manifest can be omitted, as it will be replaced anyway. See the Kubernetes documentation of resource quotas for details about Kubernetes resource quotas.

Jenkinsfile Runner

Parameter Description Default
pipelineRuns.jenkinsfileRunner.image.repository
string		 Deprecated: Use pipelineRuns.jenkinsfileRunner.image instead.
pipelineRuns.jenkinsfileRunner.image.tag
string		 Deprecated: Use pipelineRuns.jenkinsfileRunner.image instead.
pipelineRuns.jenkinsfileRunner.image.pullPolicy
string		 Deprecated: Use pipelineRuns.jenkinsfileRunner.imagePullPolicy instead.
pipelineRuns.jenkinsfileRunner.image
string		 The Jenkinsfile Runner image. stewardci/stewardci-jenkinsfile-runner:<versionTag>
pipelineRuns.jenkinsfileRunner.imagePullPolicy
string		 The image pull policy for the Jenkinsfile Runner image. For possible values see field imagePullPolicy of the container spec in the Kubernetes API documentation. IfNotPresent
pipelineRuns.jenkinsfileRunner.javaOpts
string		 The JAVA_OPTS environment variable for the Jenkinsfile Runner process. (see values.yaml)
pipelineRuns.jenkinsfileRunner.resources
object of RecourceRequirements		 The resource requirements of Jenkinsfile Runner containers. If not set or empty, the default value is used. {"limits": {"cpu":3, "memory":"2Gi"}, "requests": {"cpu":"500m", "memory":"2Gi"}}
pipelineRuns.jenkinsfileRunner.podSecurityContext.runAsUser
integer		 The user ID (UID) of the container processes of the Jenkinsfile Runner pod. The value must be an integer in the range of [1,65535]. Corresponds to field runAsUser of a PodSecurityContext. 1000
pipelineRuns.jenkinsfileRunner.podSecurityContext.runAsGroup
integer		 The group ID (GID) of the container processes of the Jenkinsfile Runner pod. The value must be an integer in the range of [1,65535]. Corresponds to field runAsGroup of a PodSecurityContext. 1000
pipelineRuns.jenkinsfileRunner.podSecurityContext.fsGroup
integer		 A special supplemental group ID of the container processes of the Jenkinsfile Runner pod, that defines the ownership of some volume types. The value must be an integer in the range of [1,65535]. Corresponds to field fsGroup of a PodSecurityContext. 1000
pipelineRuns.jenkinsfileRunner.pipelineCloneRetryIntervalSec
string		 The retry interval for cloning the pipeline repository (in seconds). The default value is defined in the Jenkinsfile Runner image.
pipelineRuns.jenkinsfileRunner.pipelineCloneRetryTimeoutSec
string		 The retry timeout for cloning the pipeline repository (in seconds). The default value is defined in the Jenkinsfile Runner image.
pipelineRuns.jenkinsfileRunner.sidecars
list		 A list of sidecar containers for the task, as specified by the Tekton documentation.

Logging

The parameters in this section configure the Elasticsearch Logs Plug-in of the Jenkinsfile Runner.

This plug-in supports two ways of sending log events:

  • Via Elasticsearch Index API: For each log event an HTTP request is send to Elasticsearch.

  • Via Fluentd Forward Protocol: This implementation is based on Fluency. Log events are first buffered in-memory. Chunks of log events are sent to a log forwarder via Fluentd Forward Protocol. The log forwarder is responsible to forward all log events to Elasticsearch (or any other type of destination). It may even apply further processing of log events.

Parameter Description Default
pipelineRuns.logging.elasticsearch.indexURL
string		 The URL of the Elasticsearch index to send logs to. If null or empty, logging to Elasticsearch is disabled. Example: http://elasticsearch-primary.elasticsearch.svc.cluster.local:9200/jenkins-logs/_doc
pipelineRuns.logging.elasticsearch.connectTimeoutMillis
integer		 For Elasticsearch API calls, the timeout in milliseconds for establishing a connection. Zero is interpreted as infinite timeout. If not set or set to a negative value, the system default is used.
pipelineRuns.logging.elasticsearch.requestTimeoutMillis
integer		 For Elasticsearch API calls, the timeout in milliseconds used when requesting a connection from the connection manager. Zero is interpreted as infinite timeout. If not set or set to a negative value, the system default is used.
pipelineRuns.logging.elasticsearch.socketTimeoutMillis
integer		 For Elasticsearch API calls, the socket timeout (SO_TIMEOUT) in milliseconds. The socket timeout is the maximum period between two consecutive data packets. Zero is interpreted as infinite timeout. If not set or set to a negative value, the system default is used.
pipelineRuns.logging.forwarder.enabled
boolean		 Whether log events should be sent to a log forwarder via Fluentd Forward Protocol instead of sending them directly to Elasticsearch. false
pipelineRuns.logging.forwarder.useSidecar
boolean		 Whether the log forwarder is running as a sidecar container of the Jenkinsfile Runner pod. If true, the pod's IP address will be used as forwarder host. false
pipelineRuns.logging.forwarder.host
string		 mandatory if forwarder.enabled is true and forwarder.useSidecar is false
The host name of the log forwarder. Ignored if useSidecar is true.
pipelineRuns.logging.forwarder.port
string		 mandatory if forwarder.enabled is true
The port the log forwarder is listening to.		 24224
pipelineRuns.logging.forwarder.tag
string		 mandatory if forwarder.enabled is true
The tag to use when sending data to the log forwarder.		 logs
pipelineRuns.logging.forwarder.senderBaseRetryIntervalMillis
integer		 The base retry interval in milliseconds used to calculate the exponential back-off when the Fluency sender retries failed operations.
pipelineRuns.logging.forwarder.senderMaxRetryIntervalMillis
integer		 The maximum retry interval in milliseconds when the Fluency sender retries failed operations.
pipelineRuns.logging.forwarder.senderMaxRetryCount
integer		 The maximum number of retries used by the Fluency sender for failed operations.
pipelineRuns.logging.forwarder.connectionTimeoutMillis
integer		 The timeout in milliseconds for establishing a connection to the log forwarder. Must be greater than 10.
pipelineRuns.logging.forwarder.readTimeoutMillis
integer		 The timeout in milliseconds for retrieving a response (acknowledgement) from the log forwarder. Must be greater than 10. Note that the value must be sufficient to process a whole chunk of log events in the log forwarder.
pipelineRuns.logging.forwarder.maxWaitSecondsUntilBufferFlushed
integer		 The timeout in seconds for shutting down the Fluency flusher service.
pipelineRuns.logging.forwarder.maxWaitSecondsUntilFlusherTerminated
integer		 The timeout in seconds for closing the Fluency buffer and ingester.
pipelineRuns.logging.forwarder.bufferChunkInitialSize
integer		 The initial size of a Fluency buffer chunk. A buffer chunks get expanded if the remaining capacity is too small for new data to be added.
pipelineRuns.logging.forwarder.bufferChunkRetentionSize
integer		 The maximum size of a single buffer chunk. Once this size would be exceeded when new data is added, the chunk gets ready to be flushed to the log forwarder and a new chunk is created. Must be greater than bufferChunkInitialSize.
pipelineRuns.logging.forwarder.bufferChunkRetentionTimeMillis
integer		 The time in seconds after which a buffer chunk will be flushed although it's not yet "full".
pipelineRuns.logging.forwarder.flushAttemptIntervalMillis
integer		 The interval in milliseconds at which the Fluency flusher service periodically checks for buffer chunks ready to be flushed.
pipelineRuns.logging.forwarder.maxBufferSize
integer		 The maximum total size in bytes of all buffer chunks. Must be greater than bufferChunkRetentionSize.
pipelineRuns.logging.forwarder.emitTimeoutMillis
integer		 The timeout in milliseconds for inserting a single log event into the local in-memory buffer and retrying in case of errors, e.g. when the buffer is full.

Helm Hooks

Images

This section configures the images used by Helm hooks.

Parameter Description Default
hooks.images.<IMAGE_KEY>.repository
string		 The container registry and repository of the image. see values.yaml
hooks.images.<IMAGE_KEY>.tag
string		 The tag of the Run Controller image in the container registry. see values.yaml
hooks.images.<IMAGE_KEY>.pullPolicy
string		 The image pull policy for the image. For possible values see field imagePullPolicy of the container spec in the Kubernetes API documentation. see values.yaml

Image keys:

  • kubectl: An image with Bourne shell and the kubectl CLI.

CRD Update Hooks

The CRD Update hooks are Kubernetes Jobs to update the Steward CRDs.

Lifecycle:

  • pre-install
  • pre-upgrade
  • pre-rollback
Parameter Description Default
hooks.crdUpdate.resources
object of RecourceRequirements		 The resource requirements of the hook container. If not set or empty, the default value is used. {}
hooks.crdUpdate.podSecurityContext
object of PodSecurityContext		 The pod security context of the hook pod. If not set or empty, the default value is used. {"runAsUser":1000, "runAsGroup":1000, "fsGroup":1000, "runAsNonRoot":true}
hooks.crdUpdate.securityContext
object of SecurityContext		 The security context of the hook container. If not set or empty, the default value is used. {"privileged":false, "seccompProfile": {"type":"RuntimeDefault"}, "allowPrivilegeEscalation":false, "capabilities": {"drop": ["ALL"]}, "readOnlyRootFilesystem":true}
hooks.crdUpdate.nodeSelector
object		 The nodeSelector field of the hook pod spec. If not set or empty, the default value is used. {}
hooks.crdUpdate.affinity
object of Affinity		 The affinity field of the hook pod spec. If not set or empty, the default value is used. {}
hooks.crdUpdate.tolerations
array of Toleration		 The tolerations field of the hook pod spec. If not set or empty, the default value is used. []

Feature Flags

Parameter Description Default
featureFlags
string		 Feature flag definition. empty

The feature flags definition is a string containing any number of feature flag names separated by any non-empty sequence of comma (,) and/or whitespace (space, horizontal tab, vertical tab, carriage return, newline, form feed). Separators at the beginning and the end of the string are allowed.

A feature flag gets enabled if the name is either prefixed with a plus sign (+) or not prefixed with a minus sign (-). A feature flag gets disabled if the name is prefixed with a minus sign (-).

Example 1: Typical format

Flag1 +Flag2 -Flag3

or alternatively using comma and space as separator:

Flag1, +Flag2, -Flag3

or alternatively using newline as separator:

Flag1
+Flag2
-Flag3

Flag1 and Flag2 get enabled, while Flag3 gets disabled.

Example 2: Untypical but correct format

, Flag1 +Flag2, ,,,,
Flag3,

It has the same effect as in example 1. The definition string has leading and trailing separators and uses different separator sequences.

List of Defined Feature Flags

Feature Flag Description Default
RetryOnInvalidPipelineRunsConfig If enabled, the pipeline run controller retries reconciling PipelineRun objects in case the controller configuration (in ConfigMaps) is invalid or cannot be loaded. It is assumed that the condition can be detected by a monitoring tool, triggers an alert and operators fix the issue in a timely manner. By that operator errors do not immediately break user pipeline runs. However, processing of PipelineRun objects may be delayed significantly in case of invalid configuration.

If disabled, the current behavior is used: immediately set all unfinished PipelineRun objects to finished with result code error_infra.

The new behavior is supposed to become the default in a future release of Steward.		 disabled

Misc

Duration Value Syntax

Chart parameters of type duration must be specified as a string understood by Go's time.parseDuration():

A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Custom Resource Definitions

Steward extends Kubernetes by a set of custom resources types like PipelineRun. The respective custom resource definitions (CRDs) are handled in a special way:

  • Upon install, upgrade and rollback the CRDs will be created or updated to the version from this chart.

    CRDs that are not part of the Steward version to be installed, upgraded to or rolled back to will NOT be deleted to prevent unexpected deletion of objects of those custom resource types.

  • An uninstall will keep the CRDs to prevent unexpected deletion of objects of those custom resource types.

  • The --force option of the helm upgrade or helm rollback command, which enables replacement by delete and recreate, does NOT apply to CRDs.

Operators may delete Steward CRDs manually after Steward has been uninstalled. By doing so, all resource objects of those types will be removed by Kubernetes, too.