DevSecOps Pipeline Demo

DevSecOps CICD pipeline demo using several technologies such as:

• OpenShift Pipelines
• OpenShift GitOps
• OpenShift Advanced Cluster Security for Kubernetes
• OpenShift Container Registry

Vulnerability and configuration management methods included in this demo are the following:

• Static application security testing (SAST), which analyzes code under development for vulnerabilities and quality issues.
• Software composition analysis (SCA), which examines dependent packages included with applications, looking for known vulnerabilities and licensing issues.
• Interactive application security testing (IAST) and dynamic application security testing (DAST) tools, which analyze running applications to find execution vulnerabilities.
• Configuration management with analysis and management of application and infrastructure configurations in DevOps. Traditionally this was not used as a way to improve security. But properly managing configurations in a GitOps process can strengthen security by improving change controls, identifying configuration defects that can reduce the attack surface, and signing and tracking authorship for better accountability and opportunities to improve.
• Image risk is any risk associated with a container image. This includes vulnerable dependencies, embedded secrets, bad configurations, malware, or images that are not trusted.

This pipeline also improve security adding the following Open Source components:

• SonarQube
• Nexus
• JUnit
• Gogs
• Git Webhook
• Gatling
• Zap Proxy

• NOTE: Tested and fully working for 4.7+ OpenShift Clusters, including >=4.9!

Overview

1. Continuous Integration

On every push to the spring-petclinic git repository on Gogs git server, the following steps are executed within the Tekton pipeline:

0. Code is cloned from Gogs git server and the unit-tests are run
1. Dependency report from the source code is generated and uploaded to the report server repository.
2. Unit tests are executed and in parallel the code is analyzed by Sonarqube for anti-patterns.
3. Application is packaged as a JAR and released to Sonatype Nexus snapshot repository
4. A container image is built in DEV environment using S2I, and pushed to OpenShift internal registry, and tagged with spring-petclinic:[branch]-[commit-sha] and spring-petclinic:latest

2. DevSecOps steps using Advanced Cluster Security for Kubernetes

Advanced Cluster Security for Kubernetes controls clusters and applications from a single console, with built-in security policies.

Using roxctl and ACS API, we integrated in our pipeline several additional security steps into our DevSecOps pipeline:

5. Image Scanning using ACS Scanner of the image generated and pushed in step 4.

6. Image Check of the build-time violations of the different security policies defined in ACS
7. Checks build-time and deploy-time violations of security policies in ACS of the YAML deployment files used for deploy our application.

NOTE: these 3 steps are executed in parallel for saving time in our DevSecOps pipeline.

8. Kubernetes kustomization files updated with the latest image [commit-sha] in the overlays for dev. This will ensure that our Application are deployed using the specific built image in this pipeline.

3. Continuous Delivery

Argo CD continuously monitor the configurations stored in the Git repository and uses Kustomize to overlay environment specific configurations when deploying the application to DEV and STAGE environments.

9. The ArgoCD applications syncs the manifests in our gogs git repositories, and applies the changes automatically into the namespaces defined:

and deploys every manifest that is defined in the branch/repo of our application:

4. PostCI - Pentesting and Performance Tests

Once our application is deployed, we need to ensure of our application is stable and performant and also that nobody can hack our application easily.

10. Our CI in Openshift Pipelines waits until the ArgoCD app is fully sync and our app and all the resources are deployed
11. The performance tests are cloned into our pipeline workspace
12. The pentesting is executed using the web scanner OWASP Zap Proxy using a baseline in order to check the possible vulnerabilities, and a Zap Proxy report is uploaded to the report server repository.
13. In parallel the performance tests are executed using the load test Gatling and a performance report is uploaded to the report server repository.

5. Notifications

ACS can be integrated with several Notifier for notify if certain events happened in the clusters managed. In our case, we integrated with Slack in order to receive notifications when some Policies are violated in order to have more useful information:

These policies notification can be enabled by each system policy enabled in our system, so you can create your own notification baseline in order to have only the proper information received in your systems.

NOTE: By now the integration is manual. WIP to automate it.

6. Image Signing and Pipeline Signing

The original demo can be extended to use Cosign to Sign Image artifacts and also to sign the Tekton Build Pipeline via Tekton Chaining.

To extend the pipeline run the extend.sh script

   ./extend.sh

This will install Noobaa(Object Storage), Quay, and create a pod for cosign secret generation and verification.It will also install the tekton chains operator and integrate with ACS policies to generate violations for non signed images.

After installation the pipeline will build images to quay and have a task that signs the image.

We also create a policy in ACS that will generate a violation for every unsigned image

Pipeline can be run normally via the Run the demo Instructions below.

After Pipeline is run Quay will show the image signed by Cosign

Since we have Tekton Chaining enabled, successfully completed Taskruns will also be annotated with cosign signatures and payload information.

And we can verify the signature and payload information of our last successful pipelinerun using the below command.

   ./demo.sh sign-verify

Security Policies and CI Violations

In this demo, we can control the security policies applied into our pipelines, scanning the images and analysing the different deployments templates used for deploy our applications.

We can enforce the different Security Policies in ACS, failing our CI pipelines if a violation of this policy appears in each step of our DevSecOps pipelines (steps 6,7,8).

This Security Policies can be defined at BUILD level (during the build/push of the image), or at DEPLOYMENT level (preventing to deploy the application).

For example this Security Policy, checks if a RH Package Manager (dnf,yum) is installed in your Image, and will FAIL the pipeline if detects that the image built contains any RH Package Manager:

This ensures that we have the total control of our pipelines, and no image is pushed into your registry or deployed in your system that surpases the Security Policies defined.

Fixing the image

To show a complete demo and show the transition from a "bad image" to an image that passes the build enforcement, we can update the Tekton task of the image build and fix the image. In this example, we will be enabling the enforcement of the "Red Hat Package Manager in Image" policy in ACS, which will fail our pipeline at the image-check as both yum and rpm package managers are present in our base image.

Update the tekton task:

1. Delete the s2i-java-11 task
   1. With the UI: From the OpenShift UI, make sure you are in the cicd project and then go to Pipelines > Tasks and delete the s2i-java-11 task.
   2. With the Tekton cli tkn task delete s2i-java-11
2. Apply the new update task: kubectl apply -f fix-image/s2ijava-mgr.yaml
3. Re-run the pipeline, your deployment now succeeds.

You can check the s2ijava-mgr.yaml file for more details. We have added a step to this Task which leverages buildah to remove the package managers from the image (search for "rpm" or "yum" in the file).

Deploy

Prerequisites

• A RHEL or Fedora box
• Openshift Cluster 4.7+
• oc binary
• Ansible 2.7+
• Git

• Install Ansible

• Install Kubernetes Ansible Module

ansible-galaxy collection install community.kubernetes
pip3 install kubernetes
pip3 install openshift

Install some extra Python dependency:

pip3 install jmespath

Bootstrap

Fully automated deployment and integration of every resource and tool needed for this demo.

oc login --token=yourtoken --server=https://yourocp

Run the installer:

./install.sh

Credentials & Resources

Check the resources deployed for this demo with:

./status.sh

• Gogs git server (username/password: gogs/gogs)
• Sonatype Nexus (username/password: admin/admin123)
• SonarQube (username/password: admin/admin)
• Argo CD (username/password: admin/[Login with OAuth using Dex])
• ACS (username/password: admin/stackrox)
• Repository Server (username/password: reports/reports)

Run the demo!

cd ..
./demo.sh start

NOTE: This pipeline will fail if you don't disable the "Fixable at least Important" policy enforcement behaviour of ACS. This is expected to demonstrate the failure when a violation of the system policy occurs.

Quick Video with the Demo

• Option I - Complete CICD End2End process (Success)

• Option II - Failure CICD pipeline due to the ACS violation policy

• Openshift Coffee Break - ACS for Kubernetes - DevSecOps Way

Promote Pipeline and Triggers

• Promote Pipeline
• Triggers in Dev Pipeline

Troubleshooting

• Check the Tshoot section

Credits

Big thanks for the contributors and reviews that helped so much in this demo! We grow as we share!