openshift-acm-oot-drv

This repository contains playbooks to manage out of tree drivers using the Special Resource Operator on Redhat ACM managed clusters.

The target deployment model is a RHACM hub cluster managing many spoke SNO clusters as illustrated below.

                                                                     ┌───────────┐
                                     ┌───────────┐                   │           ├┐
                                     │           │             ┌─────┤ SNO SET-A │├┐
                                     │ REGISTRY  │             │     │           │││
                                     │           │             │     └┬──────────┘││
                                     └───────────┘             │      └───────────┘│
                                                               │       └───────────┘
                                                               │
                                     ┌───────────┐             │
┌─────────────┐                      │           │             │     ┌───────────┐
│ Provisioner │                      │  ACM HUB  │             │     │           ├┐
│             ├──────────────────────┤           ├─────────────┼─────┤ SNO SET-B │├┐
│ Host        │                      │  Cluster  │             │     │           │││
└─────────────┘                      │           │             │     └┬──────────┘││
                                     └───────────┘             │      └───────────┘│
                                                               │       └───────────┘
                                                               │
                                                               │
                                                               │     ┌───────────┐
                                                               │     │           ├┐
                                                               └─────┤ SNO SET-C │├┐
                                                                     │           │││
                                                                     └┬──────────┘││
                                                                      └───────────┘│
                                                                       └───────────┘

The playbooks deploy SRO on the ACM Hub cluster and builds the drivers there. Then uses a MachineConfig wrapped in an ACM policy to deploy the driver.

It currently only supports Single Node Openshift (SNO) nodes as managed clusters.

Dependencies

Requires the following to be installed on the provisioner host:

• Ansible
• podman
• oc client

dnf install ansible podman pip
pip install kubernetes

There's also some required Ansible modules that can be installed with the following command:

ansible-galaxy collection install -r requirements.yml

Before Running The Playbook

There are few setup steps required prior to running the playbook with the sample driver.

• Make sure you can run the OpenShift CLI as a user with cluster-admin privileges
• Setup the necessary prerequisites by running the deploy_prerequisites.yml playbook

ansible-playbook -i inventory deploy_prerequisites.yml

• Add your pull secret as a Secret named external-registry in the oot-driver namespace so that SRO can pull/push the images it needs from/to your local registry.

# Your pull secret should contain credentials to pull from the standard image registries as well as your local registry.
export PULL_SECRET='/path/to/your/pull-secret'
oc -n oot-driver create secret generic external-registry --from-file=.dockerconfigjson=${PULL_SECRET} --type=kubernetes.io/dockerconfigjson

• Add the local registry's certificate as a ConfigMap named registry-cas in the openshift-config namespace, then update the cluster image configuration

export REGISTRY_FQDN='registry.example.lab'
export REGISTRY_PORT='5000'
export REGISTRY_CERT='/path/to/your/registry/domain.crt'

oc create configmap registry-cas -n openshift-config --from-file=${REGISTRY_FQDN}..${REGISTRY_PORT}=${REGISTRY_CERT}
oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge

• Add the specialresource.openshift.io/intel-eth-800 label that the sample driver is expecting onto the nodes that have the Intel(R) Ethernet 800 Series NIC

Running The Playbooks

The playbooks provided in this repository will build and deploy a sample driver which is for the Intel(R) Ethernet 800 Series NICs.

• deploy_sro.yml: installs SRO on the ACM HUB cluster using the provided kubeconfig
• build_drivers.yml: creates and applies the SpecialResource CRs to kick start the driver builds
• deploy_drivers.yml: deploys the driver containers onto the manged SNOs through ACM policy

To run the playbooks with the provided sample driver, use these commands.

ansible-playbook -i inventory deploy_sro.yml
ansible-playbook -i inventory build_drivers.yml
ansible-playbook -i inventory deploy_drivers.yml

Adding your own drivers

To add you own drivers, you can follow the sample driver provided.

1. Label the SNO nodes you want to deploy drivers onto. The sample driver, for example uses the specialresource.openshift.io/intel-eth-800 label to select nodes to deploy onto.

2. Create a Helm chart and the corresponding SpecialResource CR template file. The playbooks will drive your SpecialResource CR with the following variables:

• item.name: The name of the driver
• item.namespace: The namespace where to apply your SpeialResource CR
• item.drivercontainer: The driver container that will be pushed to the local registry
• item.kernelversion: The kernel version to build against
• item.dtki: The Driver Toolkit Image to use
• item.sourcecontainer: The source container from the local registry

3. Create an ACM Policy wrapped MachineConfig following the sample provided in templates/mc-ice.yml.j2

4. Add your driver to the list of managed resources in the inventory.yml file.