DevSecOps with OpenShift Platform Plus (60 mins)

We are using Gitea as our source control system for this workshop. To access Gitea, click on this URL: https://gitea-gitea.{openshift_cluster_ingress_domain}, and then login using Keycloak. Do not login directly with {user} credentials.

When you login, your Keycloak account will be automatically linked with the pre-created {user} account giving you access to the existing source code repositories as shown below:

The pre-created repositories are as follows:

We see that the pipeline hasn’t been run yet, so let’s trigger it using a "PipelineRun".

Here’s a look at the content of the pipelineRun used to trigger the pipeline. These are the main parameters that are used by the pipeline tasks to deploy the application:

This will take you to a diagram with the last pipeline execution. Let’s now examine the different steps, and focus on the tasks that provide an extra layer of security. In you want more details, you can click on each task to see the logs.

In the early stages of the pipeline, we do a traditional source clone, then we verify the code using SonarQube.

You can access your SonarQube instance to check the project using this URL below. If it’s unresponsive just wait a bit longer and refresh, because sonarqube needs the projects to be created first: https://sonarqube-{user}-cicd.{openshift_cluster_ingress_domain}/projects

Build-sign-image. Enhancing Security with Tekton Chains

This task is responsible for building a container image based from our source code, including any changes that were committed. The built container image, along with a new tag and a generated Software Bill of Materials (SBOM) is then pushed to our private quay registry on successful completion. An SBOM is a machine-readable, formally structured complete list of all the components, including modules and libraries, used/required to build a software solution. So, in simple words, a software bill of materials offers an insight into the makeup of an application developed using third-party commercial tools and open-source software.

This task also uses Tekton Chains, a Kubernetes Custom Resource Definition (CRD) controller, that is crucial in augmenting the supply chain security within our OpenShift Pipelines. This tool’s capacity to automatically sign task runs, and its adoption of advanced attestation formats like in-toto, bring a higher degree of trust and verification to our processes.

This task is responsible for emitting two important TaskResults i.e. IMAGE_URL and IMAGE_DIGEST. Those parameters are very important because they are the ones that trigger Tekton Chains to create a digital signature for your container image.

Now let’s have a look at the following tasks:

acs-image-check. This task uses the roxctl CLI to check build-time violations of your security policies in your image. In this demo, we have set up a policy that verifies signatures on your container image. If this policy is enabled and your container image is unsigned or signed by non trusted source, the pipeline will fail. If the signature is available and is trusted, this pipeline task will complete successfully.

acs-image-scan. The acs-image-scan uses the roxctl CLI to return the components and vulnerabilities found in the image. Any vulnerabilities that exist in packages embedded in the image will be reported.

scan-export-sbom. This task is responsible for scanning any vulnerabilities that exist in our SBOM and exports our SBOM to a externally accessible repository. For scanning, this task uses a 3rd-party tool called Grype which is a vulnerability scanner for container images and filesystems.

That’s it! You now have a deeper understanding of the Security capabilities that provide a Trusted Software Supply Chain (or DevSecOps approach), using OpenShift Pipelines (tekton chains), and Red Hat Advanced Cluster Security (Red Hat ACS).

We can validate the image signature with the sigstore utility called cosign. This validation can be performed in variety of ways, first let’s validate the signature manually using the public signature.

While cosign signature used to sign the image by Tekton Chains is stored in the openshift-pipelines namespace, a copy of the signature is available in the {user}-cicd namespace to support validation in the pipeline.

To view the signature run the following:

You will then see the following:

Notice there are three keys in the secret. cosign.key is the private key used to sign the image, cosign.pub is the public key used to validate the image and cosign.password is key for the password. All values are base64 encoded as per standard Kubernetes practices.

To get a local copy of the the public key we will use to validate the image execute the following:

Next verify that you retrieved the public key by viewing the file:

Finally use the cosign command to verify the image in the Quay image registry:

Red Hat Trusted Artifact Signer (RHTAS) provides a suite of tools to support image signing. These tools are primarily sourced from the upstream Sigstore project and include the following:

In this workshop RHTAS has been deployed in the cluster and Tekton Chains has been configured to automatically upload the metadata needed to verify the signature in the transparency log.

The first step is to set some environment variables so cosign will use the transparency log for verification:

Next we will initialize and configure cosign as well as retrieve certificates from the various Sigstore components required for verification with the transparency log:

Finally perform the actual verification by executing the following command:

Notice that in this iteration the existence of the signature in the transparency log was verified which was not done previously.

While manual verification is useful, ideally we want to bake this process into the Pipeline itself. This is done in the verify-tlog-signature step of the pipeline:

Click on the verify-tlog-signature task, note that an identical verification to what was performed manually has been completed:

This verification is performed every time the image is built, however in a later section we will look at how we can inform and enforce a signature requirement through a Policy. However before we do that, now that we have built the image and verified its signature, let’s deploy the front-end application.

A common pattern when deploying an application to multiple environments is to have a a repository that contains the following structure:

Let’s deploy these applications using an ApplicationSet as we did previously but this time we will use a git generator.

Let’s have a quick look at our new ApplicationSet:

Check that the Applications have been created:

We can verify that the applications have been created in the Shared OpenShift GitOps by checking the Application tiles.

If we pay closer attention, there are 3 items worth mentioning to understand the multi-environment management:

You can see more details by opening the "gitops" repository in gitea, and navigating to "globex" folder.

As mentioned in the application architecture section, our "Globex" application would be deployed across multiple namespaces or clusters using OpenShift GitOps.

Let’s explore this step in the pipeline, then have a look at argocd to understand how it uses the manifests to target the desired namespace.

Let’s now switch to OpenShift GitOps to see how the application gets deployed in the DEV namespace.

Notice that the app-dev and app-prod Applications are targeted to different namespaces, {user}-dev and {user}-prod. They could just as easily be targeted to different clusters as well since it is common in many organizations to have separate clusters for non-production versus production.

We have deployed the same application to the "PROD" environment using the app-prod Application in OpenShift GitOps. The main difference is that the prod version is using "globex/overlays/prod" for the specific values required for production.

It is common to have a "manual approval", i.e. "gating", for deploying into a production environment, and in our case, we’ll be using a Pull Request to approve the change to the PROD manifests located in "globex/overlays/prod".

The pipeline execution has already created the pull request in the last steps, so let’s review it in gitea and merge it to initiate the deployment in the "{user}-prod" namespace through GitOps.

Click on the pull request and then click on Create Merge Commit, and select create commit:

That’s it, you now have a better understanding of how the DevSecOps pipeline is combined with OpenShift GitOps for a multicluster deployment of the "Globex" application.

Red Hat ACS provides about 85 out-of-the-box policies to help implement security best practices and safeguards across your fleets of clusters, you can explore some of them by scrolling through the list of policies.

In the "Policy details" section the metadata and the Severity level and some other information is defined.

The section "Policy Behavior" is where you can define when and how the policy gets applied.

The "Lifecycle stages" allow you to define if it’s applied at Build, Deploy or Runtime. For this Policy, Build and Deploy stages have been chosen:

The response method provides 2 options:

Finally, the "Enforce" gives you control over how the policy gets enforced, as explained in the different options.

The Policy Scope section at the bottom enables you to control the scope of the policy by selecting the required clusters and namespaces. Wildcards are supported as well selection via labels.

Now that we have reviewed the policy, let’s see review the policy violations for this policy.

Let’s now see how Red Hat ACS allows you to monitor your cluster security, by inspecting image vulnerabilities.

Click on the globex-ui:main image highlighted above and this will take you to the image details, where you see a listing of all CVEs, all components, and all the deployments (in Resources tab) that are using this image. This helps you understand and mitigate issues when there’s a compromised image for example.

That’s it! You now have a better understanding of how Red Hat ACS allows you to define security policies that can in turn be used within the DevSecOps pipeline as security gates to prevent untrusted / undesirable content from getting into your production environments, and also continuously monitor the security of your multiple clusters and applications across all environments.