Skip to content

Configuration

Jump to bottom
sud82 edited this page Oct 20, 2020 · 33 revisions

Custom Resource Definition and Custom Resource

The operator CRD specifies the CR that the operator uses. The Aerospike cluster Custom Resource (CR) based on this CRD drives the deployment and management of Aerospike clusters. To create and deploy an Aerospike cluster, create a CR yaml file.

This custom resource can be edited later on to make any changes to the Aerospike cluster.

Example CR

A sample AerospikeCluster resource yaml file that sets up a persistent namespace and an in-memory namespace is below.

apiVersion: aerospike.com/v1alpha1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  build: aerospike/aerospike-server-enterprise:4.7.0.10

  rackConfig:
    namespaces:
      - test
    racks: 
      - id: 1
        zone: us-central1-b
        aerospikeConfig:
          service:
            proto-fd-max: 18000
          namespace:
            - name: test
              storage-engine:
                device:
                  - /dev/nvme0n2	/dev/sdf2
        storage:
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
          volumes:
            - path: /opt/aerospike
              storageClass: ssd
              volumeMode: filesystem
              sizeInGB: 1
            - path: /dev/nvme0n2
              storageClass: local-ssd
              volumeMode: block
              sizeInGB: 5
            - path: /dev/sdf2
              storageClass: ssd
              volumeMode: block
              sizeInGB: 5
      - id: 2
        zone: us-central1-a
        aerospikeConfig:
          service:
            proto-fd-max: 16000 

  storage:
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
    volumes:
      - path: /opt/aerospike
        storageClass: ssd
        volumeMode: filesystem
        sizeInGB: 1
      - path: /dev/nvme0n1
        storageClass: local-ssd
        volumeMode: block
        sizeInGB: 5
      - path: /dev/sdf
        storageClass: ssd
        volumeMode: block
        sizeInGB: 5

  multiPodPerHost: true

  aerospikeAccessControl:
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin

  aerospikeConfigSecret:
    secretName: aerospike-secret
    mountPath:  /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespace:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          device:
            - /dev/nvme0n1	/dev/sdf

  resources:
    requests:
      memory: 2Gi
      cpu: 200m

Other sample Aerospike Cluster CR objects can be found here

Configuration

The initial part of the CR selects the CRD and the namespace to use for the Aerospike cluster.

apiVersion: aerospike.com/v1alpha1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

Spec

The spec section provides the configuration for the cluster.

The fields are described below

Field Required Type Default Update allowed Require RollingRestart Description
size Yes Integer Yes No The size/number of Aerospike node pods to run for this cluster.
build Yes String Yes Yes The official Aerospike Enterprise Server docker image to use for the node in the cluster.
resources Yes Structure Yes Yes Configures the memory and CPU to use for the Aerospike server container.
validationPolicy No Structure Yes No Configures the custom resource validation. See Validation Policy for details.
storage No Structure Yes No Required for persistent namespaces and for Aerospike work directory, unless the validation policy skips validating persistence of the work directory. See Storage for details.
multiPodPerHost No Boolean No - Indicates if this configuration should run multiple pods per Kubernetes cluster host.
aerospikeConfigSecret No Structure Yes Yes The names of the Kubernetes secret containing files containing sensitive data like licenses, credentials, and certificates.See Aerospike Config Secret for details.
aerospikeAccessControl No Structure Yes No Required if Aerospike security is enabled. See Access Control for details
aerospikeConfig Yes configMap Yes Yes A free form configMap confirming to the configuration schema for the deployed Aerospike server version. See Aerospike Config for details.
rackConfig No Structure Yes Configures the operator to deploy rack aware Aerospike cluster. Pods will be deployed in given racks based on given configuration. See Rack Config for details.

Validation Policy

This section configures the policy for validating the cluster CR.

The fields in this structure are

Field Required Type Default Update allowed Description
skipWorkDirValidate No Boolean false Yes If true skips validating that the Aerospike work directory is stored on a persistent volume.
skipXdrDlogFileValidate No Boolean false Yes If true skips validating that the XDR digest log is stored on a persistent volume.

Storage

The storage section configures persistent volumes devices to provision and attach to the Aerospike cluster node container.

This section is required by default for persisting the Aerospike work directory. The working directory should be stored on a persistent storage to ensure pod restarts do not reset Aerospike server metadata files.

This section is also required for persisting Aerospike namespaces.

The fields in this structure are described below.

Field Required Type Default Update allowed Description
filesystemVolumePolicy No Structure Yes Volume policy for filesystem volumes
blockVolumePolicy No Structure Yes Volume policy for block volumes
Volumes No List of Structure Yes List of Volumes to attach to Aerospike pods. Cannot add or remove storage volumes dynamically

Volume Policy

Specifies persistent volumes policy to determine how new volumes are initialized.

The fields are

Field Required Type Default Update allowed Description
initMethod No Enum none Yes Controls how the volumes are initialized when the persistent volume is attached the first time to a pod. Valid values are 'none', 'dd', 'blkdiscard', 'deleteFiles'
cascadeDelete No Boolean false Yes CascadeDelete determines if the persistent volumes are deleted after the pods these volumes binds to are terminated and removed from the cluster

For filesystem volumes, initMethod can be 'none' or 'deleteFiles'. For block volumes, initMethod can be 'none', 'dd' or 'blkdiscard'.

Volume

Describes a persistent volume to be attached to Aerospike devices.

The fields are

Field Required Type Default Update allowed Description
path Yes String No The path on the pod where this block volume or filesystem volume will be attached. For block volumes, this will be the device path. For filesystem volumes, this will be the mount point.
storageClass Yes String No The name of the storage class to use.
volumeMode Yes Enum (filesystem. block) No Specified the mode this volume should be created with. Filesystem mode creates a pre-formatted filesystem and mounts it at the specified path. Block mode creates a raw device and attaches it the device path specifed above.
sizeInGB Yes Integer No The size in GB (gigabytes) to provision for this device.
initMethod No Enum none Yes Controls how this volume is initialized when the persistent volume is attached the first time to a pod. Valid values are 'none', 'dd', 'blkdiscard', 'deleteFiles'
cascadeDelete No Boolean false Yes CascadeDelete determines if the persistent volume is deleted after the pod this volume binds to is terminated and removed from the cluster

Aerospike Access Control

Provides Aerospike access control configuration for the Aerospike cluster.

Field Required Type Default Update allowed Description
roles No List of Structures Yes A list of Role structures with an entry for each role.
users N0 List of Structures Yes A list of User structures with an entry for each user. Required if Aerospike security is enabled.

If the Aerospike cluster has security enabled an entry for the "admin" user having at least "sys-admin" and "user-admin" roles is mandatory.

Aerospike Role

Configures roles to have in the Aerospike cluster.

Field Required Type Default Update allowed Description
name Yes Strings The name of this role.
privileges Yes List of Strings Yes The privileges to grant this role.

Aerospike User

Configures users to have for the aerospike cluster.

Field Required Type Default Update allowed Description
name Yes Strings No The name of this user.
secretName Yes String Yes The name of the secret containing this user's password.
roles Yes List of Strings Yes The roles to grant this user.

Aerospike Config Secret

Configures the name of the secret to use and mount path to mount the secret files on the container.

Field Required Type Default Update allowed Description
secretName Yes String Yes The name of the secret
mountPath Yes String Yes The path where the secret files will be mounted in the container.

Aerospike Config

Provides the Aerospike Server configuration to use for the server process. It is a yaml form for specifying the configuration in aerospike.conf file.

Aerospike Config .conf to yaml rules

Basic rules:

  • Sections which represent map in .conf file can be directly translated to map in .yaml file. e.g. service, security section etc.

    .conf file

    service {
  service-threads 4                
  proto-fd-max 15000
}

    .yaml file

    service:
  service-threads: 4
  proto-fd-max: 15000

  • Sections which can have multiple entry in .conf like namespace, datacenter will be translated to list of maps in .yaml file. Section_entry_name (e.g. Section_entry_name in namespace test section will be test ) will be added as map element (name: Section_entry_name) in translated map in .yaml.

    .conf file

    namespace test {
    replication-factor 2
    memory-size 4G
    storage-engine device {
            file /opt/aerospike/data/test.dat
            filesize 4G
            data-in-memory true 
    }
}
namespace bar {
    replication-factor 2
    memory-size 4G
    storage-engine memory      
}

    .yaml file

    namespace:
  - name: test
    replication-factor: 2
    memory-size: 4294967296
    storage-engine:
      file:
        - /opt/aerospike/data/test.dat
      filesize: 4294967296
      data-in-memory: true
  - name: bar
    replication-factor: 2
    memory-size: 4294967296
    storage-engine: memory

Complete sample:

aerospike.conf

service {                # Tuning parameters and process owner
    proto-fd-max 15000
}               

security {               # (Optional, Enterprise Edition only) to enable 
                         # ACL on the cluster
    enable-security true
}

logging {               # Logging configuration
    console {
        context any info
    }
    file /var/log/aerospike/aerospike.log {
        context any info
    }
}               

xdr {                    # (Optional, Enterprise Edition only) Configure
                         # Cross-Datacenter Replication
    enable-xdr true # Globally enable/disable XDR on local node.
    xdr-digestlog-path /opt/aerospike/digestlog 5G # Track digests to be shipped.
    xdr-compression-threshold 1000
    datacenter REMOTE_DC_1 {
            dc-node-address-port 172.68.17.123 3000
            dc-security-config-file /etc/aerospike/secret/security_credentials_DC1.txt
    }
}

namespace test {       # Define namespace record policies and storage engine
    enable-xdr true
    xdr-remote-datacenter REMOTE_DC_1
    replication-factor 2
    memory-size 4G
    storage-engine device {
            file /opt/aerospike/data/test.dat
            filesize 4G
            data-in-memory true # Store data in memory in addition to file.
    }
}

mod-lua {                # location of UDF modules
    user-path /opt/aerospike/usr/udf/lua  
}

aerospike.yaml

  aerospikeConfig:
    service:
      proto-fd-max: 15000

    security:
      enable-security: true
    
    logging:
      - name: console
        any: info
      - name: /var/log/aerospike/aerospike.log
        any: info
    
    xdr:
      enable-xdr: true
      xdr-digestlog-path: /opt/aerospike/xdr/digestlog 5G
      xdr-compression-threshold: 1000
      datacenter:
        - name: REMOTE_DC_1
          dc-node-address-port: "172.68.17.123 3000"
          dc-security-config-file: /etc/aerospike/secret/security_credentials_DC1.txt

    namespace:
      - name: test
        enable-xdr: true
        xdr-remote-datacenter: REMOTE_DC_1
        replication-factor: 2
        memory-size: 4294967296
        storage-engine:
          file:
            - /opt/aerospike/data/test.dat
          filesize: 4294967296
          data-in-memory: true # Store data in memory in addition to file.
    
    mod-lua:
      user-path: /opt/aerospike/usr/udf/lua

Config .yaml files are based on json schema of aerospike.conf file. Different Aerospike Server versions have may have different aerospike.conf and their json representations. These .yaml file can also be created with the help of aerospike.conf json schemas. Please check config-schemas for JSON schemas for all supported versions.

Rack Config

Configures the operator to deploy rack aware Aerospike cluster. Pods will be deployed in given racks based on given configuration. To know more about aerospike rack aware feature, See Aerospike Rack Awareness

Field Required Type Default Update allowed Require RollingRestart Description
namespaces No List of Strings Yes Yes List of Aerospike namespaces for which rack feature will be enabled.
racks Yes List of structures Yes List of racks

Rack

Rack specifies single rack config

Field Required Type Default Update allowed Update Require Rack RollingRestart Description
id Yes Integer No - Identifier for the rack.
zone No String No - Zone name for setting rack affinity. Rack pods will be deployed to given Zone.
region No String No - Region name for setting rack affinity. Rack pods will be deployed to given Region.
rackLabel No String No - Racklabel for setting rack affinity. Rack pods will be deployed in k8s nodes having rackLable aerospike.com/rack-label: <rack-label>.
nodeName No String No - K8s Node name for setting rack affinity. Rack pods will be deployed in given k8s Node.
aerospikeConfig No Structure Yes Yes This local AerospikeConfig is a patch, which will be merged recursively with common global AerospikeConfig and will be used for this Rack. See merging AerospikeConfig. If this AerospikeConfig is not given then global AerospikeConfig will be used.
storage No Structure Yes - This local Storage specify persistent storage to use for the pods in this rack. If this Storage is not given then global Storage will be used.

Merging AerospikeConfig

Local rack AerospikeConfig patch will be merged with common global base AerospikeConfig using given rules.

  • New elements from the patch configMap then it will be added in base configMap
  • Base element will be replaced with new patch element if
    • Element value type is changed
    • Element value is primitive type and updated
    • Element value is primitive list type and updated
    • Element key is storage-engine and its storage-engine type has been changed. (storage-engine can be of device, file and memory type.
  • If element are of map type then patch and base elements will be recursively merged
  • If element are list of map then new list elements in patch list will be appended to base list and corresponding entries will be merged using the same merge algorithm. Here order of elements in base list will be maintained. (corresponding etries are found by matching special name key in maps. Here this list of map is actually a map of map and main map keys are added in sub-map with key as name to convert map of map to list of map).

e.g.

Rack local aerospikeConfig and common global aerospikeConfig

  rackConfig:
    racks: 
        aerospikeConfig:
          service:
            proto-fd-max: 18000
          namespace:
            - name: test
              storage-engine:
                device:
                  - /dev/nvme0n2	/dev/sdf2
            - name: bar
              memory-size: 6000000000
              storage-engine: memory
.
.
.
  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespace:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          device:
            - /dev/nvme0n1	/dev/sdf
      - name: bar
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          device:
            - /dev/nvme0n10	/dev/sdf10

After merging rack local aerospikeConfig

  aerospikeConfig:
    service:
      proto-fd-max: 18000
      feature-key-file: /etc/aerospike/secret/features.conf
    security:
      enable-security: true
    namespace:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        # storage-engine type is not changed hence its merged recursively
        storage-engine:
          device:
            - /dev/nvme0n2	/dev/sdf2
      - name: bar
        memory-size: 6000000000
        replication-factor: 2
        # storage-engine type is changed hence its replaced
        storage-engine: memory

Next

Clone this wiki locally