Advanced OpenShift GitOps(30 mins)

By default the OpenShift GitOps operator will install an instance in the openshift-gitops namespace, this instance is cluster-scoped and users are encouraged to use this instance for cluster configuration including managing privileged resources on behalf of tenants (Namespaces, Quotas, Operators, etc).

Lets have a brief look at this instance first, to do so access the OpenShift Application Menu and select the Cluster Argo CD item highlighted in red:

This will take you to the Argo CD user interface for this instance, to login click the Keycloak button shown in red:

Assuming you have already logged into OpenShift with Keycloak you will be automatically logged into Argo CD since we are using Keycloak to provide Single-Sign-On in this workshop. Once you are logged in you will see the following applications:

Notice how you can view all of the user projects (user1, user2, user3, etc), this is because we have used Argo CD RBAC to provide users in the developers group with limited access to the Argo CD Project that these applications are tied to.

Select the app for your particular user called {user}. Depending on the number of users attending the workshop there could be a lot of Applications, you can find your application by using the filter bar:

Once you find the Application, click on it to view all of the resources that this Application deployed. This Application is provisioning all of the resources that a tenant, i.e. your user, requires on this cluster including namespaces, rolebindings, etc. These are the resources that the platform team, i.e. cluster-admins, need to provision for developer teams on the cluster.

You can view the groups associated with your logged in user by clicking on the user button.

Note the developers group that is shown along with the user specific team, teamX, that your user is associated with. In our scenario user1 is on team 1, user2 on team2, etc. The developers group is a generic group that all of the application developers belong to.

These groups are provisioned in Keycloak and made available to Argo CD when you logged into the application.

In the next section we will review the Shared Argo CD instance and do a deeper dive afterwards on how Argo CD RBAC is configured to determine the access that users have to resources in Argo CD. For now know that you have very limited permissions on the Applications in the Users project as only Get and Sync permissions have been enabled.

The Shared GitOps instance is the multi-tenant Argo CD instance that developer teams, i.e. you, will interact with to deploy applications and resources. Compared to the Cluster GitOps instance it has much less privileges on the cluster to align with developer team responsibilities around deploying applications.

Login into the Shared GitOps instance by using the link in the OpenShift application menu:

The login process is identical to the Cluster GitOps instance, click the Keycloak login button to complete the process. Once logged in a set of Applications will be displayed:

Note in the Shared GitOps, unlike the Cluster GitOps instance, you can only view Applications that are specific to your user.

We have a dev-tools Application, along with the other applications deployed in the previous module, to deploy team specific Sonarqube and Nexus instances which will be used in the Pipelines module of this workshop. If you check with your neighbors, assuming they have reached this section, you will note that every person has their own dev-tools Application which is unique to that workshop user. How can we have an Application with the same name multiple times in the same Argo CD instance?

To understand this, we are going to look at this and Argo CD RBAC in more depth in the subsequent sections.

We wil use the ArgoCD CLI to explore the Shared GitOps in more detail. A secret has been pre-created in the {user}-argocd namespace called argocd-cli that provides the credentials needed to login into Argo CD.

To provision secret into the terminal as exported environment variables run the following commands:

The Argo CD CLI will use the specified environment variables automatically and not require an explicit login. Additionally the alias command at the end will ensure that when the argocd is called the parameter --grpc-web is automatically added. Since we are routing commands through the OpenShift Route this parameter is needed to avoid superfluous warnings.

Test the variables are set by using the Argo CD CLI to view the Applications that were shown in the user interface:

The following output will be provided showing the application name, the sync and health status, the source and destination.

A detailed view of the Application can be retrieved by using the get command:

Various details of the Application are shown including a list of resources that the application is managing and their associated statuses.

In addition to retrieving information about the Application, various tasks can be performed via the CLI including syncing, refreshing and modifying the Application. We will look at these in more depth in subsequent sections.

Argo CD Projects are used to group Applications together as well as manage permissions to the Applications and other Project scoped resources. Keep in mind that an Argo CD Project is different then an OpenShift Project despite using the same terminology. An OpenShift Project is represented by kind: Project in Kubernetes whereas an Argo CD Project is represented by kind: AppProject.

While every Application in Argo CD must be associated with a Project, they are particularly useful when managing a multi-tenant Argo CD as the Project not only determines the user permissions but can also restrict what Applications associated with the Project can do. As per the documentation, an Argo CD Project can:

In our Shared GitOps instance each workshop team, and thus user, has their own Project to manage access and restrictions for their Applications. To view your teams project, use the CLI to run the following command:

Notice that a single project is listed:

A detailed view of the project is retrieved by using the get command:

Notice that your user’s Applications are limited to deploying to a specific set of namespaces, the {user}-* namespaces. This limitation on destinations ensures that Applications in this Project cannot deploy resources to other namespaces even though the underlying Argo CD application-controller has permissions to do so.

Also note that we are denying access to some types of resources. All cluster scoped resources, i.e. resources without a namespace, are denied. Also note that we are denying access to specific namespace scoped resources such as ResourceQuota and LimitRange because it is the purview of the platform team to manage these resources.

Finally some Argo CD resources, such as clusters and repositories, can be scoped globally or at a Project level. Scoping resources at a Project level can be useful in cases where the Argo CD administrator would like to enable self-service for application teams. In this workshop these resources are defined globally however if you would like to learn more about this capability the Argo CD documentation covers this topic in depth.

Project scoped Repositories and Clusters

Argo CD has it’s own Role Based Access Control (RBAC) that is separate and distinct from Kubernetes RBAC. When users interact with Argo CD via the Argo CD UI, CLI or API the Argo CD RBAC is enforced. If users interact with Argo CD resources directly using the OpenShift Console or kubectl/oc then only the Kubernetes RBAC is used.

Additionally the application-controller in Argo CD, as shown previously in the GitOps Architecture, interacts with the Kubernetes API and is governed by Kubernetes RBAC. Argo CD can only deploy and manage the Kubernetes resources that the application-controller has been given permission to use in Kubernetes RBAC.

This relationship is shown in the following diagram:

The Argo CD RBAC is implemented using the Casbin library. Permissions are defined by creating roles and then assigning those roles to groups, or individual users, as needed. Argo CD includes two roles out of the box:

Roles and permissions can be defined in two places, globally and on a per Project basis. It is strongly recommended that tenant roles and permissions be defined in the Project and global roles be reserved for Argo CD administrators and managing globally scoped resources.

As a developer, i.e. user of the workshop, you do not have access to view the global configuration, however this is what is defined:

A few items to note about this global configuration:

Now let’s look at the RBAC defined in the Project that has been setup for your team and user:

This shows that two roles are defined, admin and pipeline. The admin role is intended for users who will administer Applications in this Project. The pipeline role is intended for automation tools and will be used by OpenShift Pipelines in a later module.

Now look at how the roles are defined:

Here we can see information about the role including policies, let’s break down the first policy into it’s constituent parts to understand how it is defined.

Once we have defined our role via policies we can then assign the policy to a group, this is indicated by a g at the start of the line. For example, the line g, teamX, proj:{user}:admin indicates we are assigning the project scoped role admin to the group teamX.

Note that what is considered a group for matching purposes is controlled by the scope that was reviewed earlier. While groups is the most commonly set scope, having scopes like email allows you to match roles to individual users. In a nutshell, when adding additional scopes, like email, these are treated as groups by Argo CD for matching purposes.

As discussed, the ability to view this basic project has been granted but not to modify or delete it. To confirm that, try deleting the Project:

As expected we were denied permission to delete the Project. Again as a reminder the Argo RBAC is only used when you interact with Argo CD via its UI or CLI. If you have permissions on the AppProject kind in the Argo CD namespace you could delete the resource with oc delete appproject {user} -n gitops and the Argo CD RBAC would never be checked,

This inability to let users interact with resources in the Argo CD namespace is a challenge when you want to give users the ability to declaratively manage Applications instead of managing them imperatively via the UI or CLI, fortunately Argo CD has you covered as we will see in the next module.

You may have noticed a subtle difference between the Cluster and Shared instancs of GitOps in that the Application names in the shared instance are prefixed with a namespace where as the Cluster Applications are not.

Run the following command again:

The following output will be provided, notice that the Application name consists of two parts using the format <Namespace>/<Application Name>:

The reason why is that we are using the Apps in Any Namespace feature. Traditionally, Argo CD requires that Application resources be in the same namespace as the Argo CD installation. Apps in Any Namespace allow applications to exist in other namespaces and provides a mechanism to isolate team’s Applications in a multi-tenant Argo CD instance.

This addresses an important limitation of having all Applications in the same namespace where users cannot be allowed to declaratively manage their Application resources. If they were permitted to do so, they could easily bypass the security provided by Argo CD Projects simply by assigning any Project to the Application by creating or modifying it using Kubernetes APIs (i.e. oc apply, oc edit). This problem arises because Argo CD RBAC is only enforced when using the Argo UI or CLI, it is not enforced by Kubernetes.

The Apps in Any Namespace feature avoids this issue by requiring the platform team to bind a specific Argo CD Project to each specific namespace where Applications reside. In this workshop this means the user1-argocd namespace is automatically bound to Project user1, user2-argocd namespace to user2, etc.

The namespace being included as part of the Application name is a subtle indication that Apps in Any Namespace is being used

To configure Apps in Any Namespace there are two places where the source namespaces for Applications must be configurd. The first is as a startup parameter for the Argo CD instance, with the OpenShift GitOps operator this is determined by the sourceNamespaces field in the ArgoCD CR. Use the following command to see additional information about the field:

This field takes an array of namespaces however wildcards can be used to reduce management effort. It is highly recommended when using Apps in Any Namespace to select a consistent naming pattern for these namespaces that is amenable to wildcards.

The second place to configure sourceNamespaces is in the Argo CD Project. This determines which namespaces, and correspondingly the Applications in that namespace, are associated with which Project. This ensures that when a user creates or modifies an Application in namespace X that it is always associated with Project Y even if the user tries to bypass security and select a different Project.

At the moment the Argo CD CLI does not support displaying the sourceNamespaces however they can be viewed in the UI. Switch to the Argo CD UI and navigate to "Settings > Projects" and select project {user}. You should see the sourceNamespaces defined as follows: