Skip to content

Configuration

Ashish Shinde edited this page Nov 27, 2020 · 33 revisions

Custom Resource Definition and Custom Resource

The operator CRD specifies the CR that the operator uses. The Aerospike cluster Custom Resource (CR) based on this CRD drives the deployment and management of Aerospike clusters. To create and deploy an Aerospike cluster, create a CR yaml file.

This custom resource can be edited later on to make any changes to the Aerospike cluster.

Example CR

A sample AerospikeCluster resource yaml file that sets up a persistent namespace and an in-memory namespace is below.

apiVersion: aerospike.com/v1alpha1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:4.8.0.6
  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        # Change to the zone for your k8s cluster.
        zone: us-west1-a
        # nodeName: kubernetes-minion-group-qp3m
        aerospikeConfig:
          service:
            proto-fd-max: 18000
        # rack level storage, used by pods of this rack
        storage:
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
          volumes:
            - storageClass: ssd
              path: /opt/aerospike
              volumeMode: filesystem
              sizeInGB: 1
            - path: /opt/aerospike/data
              storageClass: ssd
              volumeMode: filesystem
              sizeInGB: 3
      - id: 2
        # Change to the zone for your k8s cluster.
        zone: us-west1-a
        # nodeName: kubernetes-minion-group-tft3
        aerospikeConfig:
          service:
            proto-fd-max: 16000

  storage:
    filesystemVolumePolicy:
      cascadeDelete: true
      initMethod: deleteFiles
    volumes:
      - storageClass: ssd
        path: /opt/aerospike
        volumeMode: filesystem
        sizeInGB: 1
      - path: /opt/aerospike/data
        storageClass: ssd
        volumeMode: filesystem
        sizeInGB: 3

  multiPodPerHost: true

  aerospikeAccessControl:
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin

  aerospikeConfigSecret:
    secretName: aerospike-secret
    mountPath:  /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespaces:
      - name: test
        memory-size: 3000000000
        replication-factor: 1
        storage-engine:
          files:
            - /opt/aerospike/data/test.dat
          filesize: 2000000000
          data-in-memory: true
      - name: testMem
        memory-size: 3000000000
        replication-factor: 1
        storage-engine: memory

  resources:
    requests:
      memory: 2Gi
      cpu: 200m

Other sample Aerospike Cluster CR objects can be found here

Configuration

The initial part of the CR selects the CRD and the namespace to use for the Aerospike cluster.

apiVersion: aerospike.com/v1alpha1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

Spec

The spec section provides the configuration for the cluster.

The fields are described below

Field Required Type Default Update allowed Require RollingRestart Description
size Yes Integer Yes No The size/number of Aerospike node pods to run for this cluster.
image Yes String Yes Yes The official Aerospike Enterprise Server docker image to use for the node in the cluster.
resources Yes Structure Yes Yes Configures the memory and CPU to use for the Aerospike server container.
validationPolicy No Structure Yes No Configures the custom resource validation. See Validation Policy for details.
storage No Structure Yes No Required for persistent namespaces and for Aerospike work directory, unless the validation policy skips validating persistence of the work directory. See Storage for details.
multiPodPerHost No Boolean No - Indicates if this configuration should run multiple pods per Kubernetes cluster host.
aerospikeConfigSecret No Structure Yes Yes The names of the Kubernetes secret containing files containing sensitive data like licenses, credentials, and certificates.See Aerospike Config Secret for details.
aerospikeAccessControl No Structure Yes No Required if Aerospike security is enabled. See Access Control for details
aerospikeConfig Yes configMap Yes Yes A free form configMap confirming to the configuration schema for the deployed Aerospike server version. See Aerospike Config for details.
rackConfig No Structure Yes Configures the operator to deploy rack aware Aerospike cluster. Pods will be deployed in given racks based on given configuration. See Rack Config for details.

Validation Policy

This section configures the policy for validating the cluster CR.

The fields in this structure are

Field2 Required Type Default Update allowed Description
skipWorkDirValidate No Boolean false Yes If true skips validating that the Aerospike work directory is stored on a persistent volume.
skipXdrDlogFileValidate No Boolean false Yes If true skips validating that the XDR digest log is stored on a persistent volume.

Storage

The storage section configures persistent volumes devices to provision and attach to the Aerospike cluster node container.

This section is required by default for persisting the Aerospike work directory. The working directory should be stored on a persistent storage to ensure pod restarts do not reset Aerospike server metadata files.

This section is also required for persisting Aerospike namespaces.

The fields in this structure are described below.

Field Required Type Default Update allowed Description
filesystemVolumePolicy No Structure Yes Volume policy for filesystem volumes
blockVolumePolicy No Structure Yes Volume policy for block volumes
Volumes No List of Structure Yes List of Volumes to attach to Aerospike pods. Cannot add or remove storage volumes dynamically

Volume Policy

Specifies persistent volumes policy to determine how new volumes are initialized.

The fields are

Field Required Type Default Update allowed Description
initMethod No Enum none Yes Controls how the volumes are initialized when the persistent volume is attached the first time to a pod. Valid values are 'none', 'dd', 'blkdiscard', 'deleteFiles'
cascadeDelete No Boolean false Yes CascadeDelete determines if the persistent volumes are deleted after the pods these volumes binds to are terminated and removed from the cluster

For filesystem volumes, initMethod can be 'none' or 'deleteFiles'. For block volumes, initMethod can be 'none', 'dd' or 'blkdiscard'.

Volume

Describes a persistent volume to be attached to Aerospike devices.

The fields are

Field Required Type Default Update allowed Description
path Yes String No The path on the pod where this block volume or filesystem volume will be attached. For block volumes, this will be the device path. For filesystem volumes, this will be the mount point.
storageClass Yes String No The name of the storage class to use.
volumeMode Yes Enum (filesystem. block) No Specified the mode this volume should be created with. Filesystem mode creates a pre-formatted filesystem and mounts it at the specified path. Block mode creates a raw device and attaches it the device path specifed above.
sizeInGB Yes Integer No The size in GB (gigabytes) to provision for this device.
initMethod No Enum none Yes Controls how this volume is initialized when the persistent volume is attached the first time to a pod. Valid values are 'none', 'dd', 'blkdiscard', 'deleteFiles'
cascadeDelete No Boolean false Yes CascadeDelete determines if the persistent volume is deleted after the pod this volume binds to is terminated and removed from the cluster

Aerospike Access Control

Provides Aerospike access control configuration for the Aerospike cluster.

Field Required Type Default Update allowed Description
roles No List of Structures Yes A list of Role structures with an entry for each role.
users N0 List of Structures Yes A list of User structures with an entry for each user. Required if Aerospike security is enabled.

If the Aerospike cluster has security enabled an entry for the "admin" user having at least "sys-admin" and "user-admin" roles is mandatory.

Aerospike Role

Configures roles to have in the Aerospike cluster.

Field Required Type Default Update allowed Description
name Yes Strings The name of this role.
privileges Yes List of Strings Yes The privileges to grant this role.

Aerospike User

Configures users to have for the aerospike cluster.

Field Required Type Default Update allowed Description
name Yes Strings No The name of this user.
secretName Yes String Yes The name of the secret containing this user's password.
roles Yes List of Strings Yes The roles to grant this user.

Aerospike Config Secret

Configures the name of the secret to use and mount path to mount the secret files on the container.

Field Required Type Default Update allowed Description
secretName Yes String Yes The name of the secret
mountPath Yes String Yes The path where the secret files will be mounted in the container.

Aerospike Config

The YAML form of Aerospike server configuration. See Aerospike Configuration for detail.

Rack Config

Configures the operator to deploy rack aware Aerospike cluster. Pods will be deployed in given racks based on given configuration.

Field Required Type Default Update allowed Require RollingRestart Description
namespaces No List of Strings Yes Yes List of Aerospike namespaces for which rack feature will be enabled.
racks Yes List of structures Yes List of racks

See Rack aware for details.

Rack

Rack specifies single rack config

Field Required Type Default Update allowed Update Require Rack RollingRestart Description
id Yes Integer No - Identifier for the rack.
zone No String No - Zone name for setting rack affinity. Rack pods will be deployed to given Zone.
region No String No - Region name for setting rack affinity. Rack pods will be deployed to given Region.
rackLabel No String No - Racklabel for setting rack affinity. Rack pods will be deployed in k8s nodes having rackLable aerospike.com/rack-label: <rack-label>.
nodeName No String No - K8s Node name for setting rack affinity. Rack pods will be deployed in given k8s Node.
aerospikeConfig No Structure Yes Yes This local AerospikeConfig is a patch, which will be merged recursively with common global AerospikeConfig and will be used for this Rack. See merging AerospikeConfig. If this AerospikeConfig is not given then global AerospikeConfig will be used.
storage No Structure Yes - This local Storage specify persistent storage to use for the pods in this rack. If this Storage is not given then global Storage will be used.

Merging AerospikeConfig

Local rack AerospikeConfig patch will be merged with common global base AerospikeConfig using given rules.

  • New elements from the patch configMap then it will be added in base configMap
  • Base element will be replaced with new patch element if
    • Element value type is changed
    • Element value is primitive type and updated
    • Element value is primitive list type and updated
    • Element key is storage-engine and its storage-engine type has been changed. (storage-engine can be of device, file and memory type.
  • If element are of map type then patch and base elements will be recursively merged
  • If element are list of map then new list elements in patch list will be appended to base list and corresponding entries will be merged using the same merge algorithm. Here order of elements in base list will be maintained. (corresponding etries are found by matching special name key in maps. Here this list of map is actually a map of map and main map keys are added in sub-map with key as name to convert map of map to list of map).

e.g.

Rack local aerospikeConfig and common global aerospikeConfig

  rackConfig:
    racks:
        aerospikeConfig:
          service:
            proto-fd-max: 18000
          namespace:
            - name: test
              storage-engine:
                device:
                  - /dev/nvme0n2	/dev/sdf2
            - name: bar
              memory-size: 6000000000
              storage-engine: memory
.
.
.
  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespace:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          device:
            - /dev/nvme0n1	/dev/sdf
      - name: bar
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          device:
            - /dev/nvme0n10	/dev/sdf10

After merging rack local aerospikeConfig

  aerospikeConfig:
    service:
      proto-fd-max: 18000
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespace:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        # storage-engine type is not changed hence its merged recursively
        storage-engine:
          device:
            - /dev/nvme0n2	/dev/sdf2
      - name: bar
        memory-size: 6000000000
        replication-factor: 2
        # storage-engine type is changed hence its replaced
        storage-engine: memory

Next