Use OperatorPolicy to manage Kubernetes-native applications

You can use the policy framework in Red Hat Advanced Cluster Management for Kubernetes (RHACM) to define and manage configurations of resources across a fleet of clusters. When packaged with the Operator Lifecycle Manager (OLM), Kubernetes-native applications become resources that you can manage, similar to the built-in resource types on a cluster. In the 2.10 release of RHACM, we are introducing a new kind of policy template, OperatorPolicy, as a tech preview feature which helps manage these specific types more completely and more easily.

In this article, you’ll learn the basics of the OperatorPolicy, and you’ll see a demo about how to use it. Because this is a tech preview feature, we are looking forward to hearing your feedback about how we can improve these features for you.

Update: this feature is generally available in the 2.11 release of RHACM, and introduces it with its new features. Learn more here.

OperatorPolicy overview

As a new kind of policy template, the primary way of distributing an OperatorPolicy to a cluster is by defining it inside a policy, just like ConfigurationPolicies. The policy can then be associated with a Placement and PlacementBinding in order to propagate it to a managed cluster. To learn more about this process, see our other posts. In this article, we will focus on a specific example of just the OperatorPolicy. See the following example:

apiVersion: policy.open-cluster-management.io/v1beta1
kind: OperatorPolicy
metadata:
  name: quay-operator
spec:
  remediationAction: inform
  severity: high
  complianceType: musthave
  operatorGroup: # optional
    name: quay-opgroup
    namespace: policy-blog
    targetNamespaces:
        - policy-blog
  subscription:
    channel: stable-3.10
    name: quay-operator
    namespace: policy-blog
    installPlanApproval: Manual
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    startingCSV: quay-operator.v3.10.0
  versions:
    - quay-operator.v3.10.0

Some of the spec fields should be familiar to users of ConfigurationPolicy. These fields have a similar purpose in an OperatorPolicy:

• You can set the remediationAction field to inform or enforce. When set to enforce, the controller on the managed cluster can take actions to move the policy towards compliance. In particular, it can create resources or approve InstallPlans. When set to inform, the policy only observes and reports on the current status and does not take any actions on the cluster.
• You can set the severity field can be set to low, medium, high, or critical.
• The complianceType field currently only supports musthave, indicating the policy wants the operator described in the policy to exist.

The remaining fields are specific to OperatorPolicy:

• The operatorGroup field is optional, and is used to define details needed for the OperatorGroup. The configuration here limits the scope of the operator to a single namespace called policy-blog. For any fields that you want to put in the spec of an OperatorGroup, you can put in this field, in addition to the name and namespace. When this field is unset, the policy accepts any existing OperatorGroup in the namespace, and can create a default AllNamespaces group if the policy is being enforced. 
• The subscription field defines the operator, generally specifying what the policy wants installed and where to look for updates. Any fields available on a Subscription spec can be set here. The example here is for a specific version of the quay-operator.
• The versions field is optional, and takes a list of allowed versions of the operator. If not set, any version of the operator can be considered compliant. When this field is set, the installed version on the cluster must exactly match a version in the list in order for the policy to be compliant. When the policy is enforced, it approves updates to the operator that match a version in this list.

Viewing the status during an upgrade

Time to see an OperatorPolicy in action! Although the status of an OperatorPolicy is designed to be structured and readable directly by users, it’s helpful to view it in the RHACM console. After creating the example policy, we can view the related resources section in the web UI, as shown in Figure 1.

Because the remediationAction of the policy is set to inform, and the operator is not yet installed on the cluster, the policy is reported as “NonCompliant.” In the details, we can see that the Subscription and the OperatorGroup are currently missing, and those are both listed as reasons that the policy is NonCompliant. However, the CatalogSource used for the operator is present and healthy on the cluster. We can even click on that View YAML link to view more details about that resource on the managed cluster.

By enforcing the policy, we can have the controller on the managed cluster create the Subscription that we want for the operator, and then the UI is updated; for example, see Figure 2.

We can see that the Subscription and OperatorGroup were created based on the content in the policy, and an initial InstallPlan has been completed. The policy also reports on the Deployment and ClusterServiceVersion for the operator, and although this is not demonstrated here, if the InstallPlan or the Deployment had an error, that information would be surfaced in the policy status. Again, we can use the View YAML links to view the full content of these resources on the managed cluster.

The policy is still NonCompliant because there is an upgrade available for the operator. The policy controller does not automatically approve that new InstallPlan, because it does not match a version specified in the OperatorPolicy spec.versions list. In the UI, we can also view the policy's history (Figure 3).

This view makes it clear that the initial installation of the operator succeeded to version "quay-operator.v3.10.0", and the policy waited for the operator deployment to become healthy before becoming Compliant. Soon after that, OLM created a new InstallPlan to update the operator to version "quay-operator.v3.10.4", at which point the policy became NonCompliant again.

If we edit the policy to add this new version to the allowed spec.versions list, then the controller approves the InstallPlan for us. This process allows you to manage operator versions across many clusters, which might use different catalog sources, and therefore, have different versions available at different times. Approving individual InstallPlans like this would be very cumbersome with ConfigurationPolicies, likely requiring advanced templating and possibly not always working on all clusters. We hope that this feature provides you with a useful tool for more control than the usual "Automatic" option provided by OLM, without the burden of manually approving multiple InstallPlans across a fleet of clusters.

After adding the new version to the policy, we can view the related resources again (Figure 4).

The policy is now Compliant. Notice that the "No Status" messages attached to the InstallPlans do not indicate that status information is missing. Instead, it refers to the fact that these InstallPlans do not affect the compliance of the policy. Previously completed or failed InstallPlans are usually retained for historical information and do not necessarily indicate the status of the current installation.

Tech preview status

As mentioned, the OperatorPolicy kind is available in Red Hat Advanced Cluster Management for Kubernetes 2.10 as a tech preview feature. Details in this article are likely to change as the feature continues to develop. In particular, we have already identified certain status messages that are more noisy than useful, and that different users could benefit from some additional options around available upgrade reporting. Few people would want to be paged out early in the morning for a NonCompliant OperatorPolicy just because a new version is available.

We have several more features that we are hoping to implement in OperatorPolicy to make the process of installing and managing operators easier for new users, who might be less familiar with all the details and complexity of the Operator Lifecycle Manager. For Red Hat OpenShift users, we hope to integrate our Create Policy wizard with the operator catalog on the hub cluster, for better discovery of operators that can be installed.

Anyone interested can view the design document/enhancement proposal in the community. We welcome any and all feedback!