How oc-mirror version 2 enables disconnected installations in OpenShift 4.16

The command line utility oc-mirror is an important tool that enables cloud administrators to perform Red Hat OpenShift installations in disconnected and/or air gapped environments. 

The newer version of oc-mirror, v4.16, includes re-engineered functionality as part of version 2 (v2), developed to address challenges faced in version 1 (v1), and aims to ensure a seamless experience for end users while using oc-mirror with extra consideration for enclave support. Currently this is in technology preview for OpenShift 4.16 with plans of making it generally available in a later version.

This guide demonstrates the use of oc-mirror v2 to assist in populating a local Red Hat Quay registry that will be used for a disconnected installation, and includes the steps used to configure openshift-marketplace to use catalog sources that point to the local Red Hat Quay registry.

What this guide is

• An exploration of a technical preview feature.
• A proof of concept.
• A demonstration of a disconnected install of single node OpenShift cluster (v4.16) using local Red Hat Quay registry (standalone v3.8.14) in a home lab.

What this guide is not

• A production ready solution.
• A recommendation to switch immediately to oc-mirror v2.

Populating your local registry

To get access to the latest oc-mirror version (to ensure it includes v2), visit this link and choose either oc-mirror.tar.gz or oc-mirror.rhel9.tar.gz.

Get your OpenShift pull secret

Get your pull secret here, and save it in a file name (e.g., pull_secret.txt). We will need to add authentication for the local registry (e.g., registry.local.momolab.io:8443).

To make the pull secret easier to work with (as we want to customize it), use jq to rearrange the pull secret:

$ cat pull_secret.txt | jq . > custom_pull_secret.txt

Let’s assume your registry user is quayadmin and the password is mypassword. Then the auth entry will look like this:

$ echo  "quayadmin:mypassword" | base64 
cXVheWFkbWluOm15cGFzc3dvcmQK

Test the string by decoding it:

$ echo cXVheWFkbWluOm15cGFzc3dvcmQK | base64 -d
quayadmin:mypassword

Now, your customized pull secret (custom_pull_secret.txt) will contain something like this:

    "registry.local.momolab.io:8443": {
      "auth": "cXVheWFkbWluOm15cGFzc3dvcmQK",
      "email": "quayadmin@registry.local.momolab.io"
    },

Insert this entry just after:

{
  "auths": {

And before:

  "quay.io": {

Now this pull secret is ready to be used as either a Podman auth.json file or a Docker config file. For Podman, the location of this file would be: 

$XDG_RUNTIME_DIR/containers/auth.json

And for Docker:

~/.docker/config.json

Note that oc-mirror will use the Podman auth.json.

Determine what you want to mirror via Imageset Config

This is the example used:

$ cat imageset-config-ocmirrorv2-v4.16.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.16
      minVersion: 4.16.17
      maxVersion: 4.16.17
      type: ocp
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    full: false
  - catalog: registry.redhat.io/redhat/certified-operator-index:v4.16
    full: false
  - catalog: registry.redhat.io/redhat/community-operator-index:v4.16
    full: false
  additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest
  - name: registry.redhat.io/openshift4/ose-cli:latest

 Get registry certificates:

$ openssl s_client -connect registry.local.momolab.io:8443 -showcerts | awk '/BEGIN/,/END/{print $0}' | sudo tee ./quay-ocp.pem
$ sudo cp ./quay-ocp.pem /etc/pki/ca-trust/source/anchors/
$ sudo update-ca-trust

Add the certificate in quay-ocp.pem to your additionalTrustBundle. You can add indentation for it to fit in your YAML file using this command:

$ sed "s/^/  /" ./quay-ocp.pem

You should see something like this:

additionalTrustBundle: |
 -----BEGIN CERTIFICATE-----
 ..
 -----END CERTIFICATE-----

Running oc-mirror v2:

$ oc-mirror --v2 -c imageset-config-ocmirrorv2-v4.16.yaml  --loglevel debug   --workspace file:////data/oc-mirror/workdir/   docker://registry.local.momolab.io:8443/mirror 2>&1 | tee oc-mirror-v2-logs-202400904-debug.txt 

After the completion of this command, you should see the following output: 

2024/09/04 10:30:19  [INFO]   : 📄 Generating IDMS file...
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources/idms-oc-mirror.yaml does not exist, creating it
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources dir created
2024/09/04 10:30:19  [INFO]   : /data/oc-mirror/workdir/working-dir/cluster-resources/idms-oc-mirror.yaml file created
2024/09/04 10:30:19  [INFO]   : 📄 Generating ITMS file...
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources/itms-oc-mirror.yaml does not exist, creating it
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources dir created
2024/09/04 10:30:19  [INFO]   : /data/oc-mirror/workdir/working-dir/cluster-resources/itms-oc-mirror.yaml file created
2024/09/04 10:30:19  [INFO]   : 📄 Generating CatalogSource file...
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-certified-operator-index-v4-16.yaml does not exist, creating it
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources dir created
2024/09/04 10:30:19  [INFO]   : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-certified-operator-index-v4-16.yaml file created
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-community-operator-index-v4-16.yaml does not exist, creating it
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources dir created
2024/09/04 10:30:19  [INFO]   : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-community-operator-index-v4-16.yaml file created
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-redhat-operator-index-v4-16.yaml does not exist, creating it
2024/09/04 10:30:19  [DEBUG]  : /data/oc-mirror/workdir/working-dir/cluster-resources dir created
2024/09/04 10:30:19  [INFO]   : /data/oc-mirror/workdir/working-dir/cluster-resources/cs-redhat-operator-index-v4-16.yaml file created
2024/09/04 10:30:19  [INFO]   : mirror time     : 5h43m42.740939657s
2024/09/04 10:30:19  [WARN]   : [Worker] some errors occurred during the mirroring.
 Please review /data/oc-mirror/workdir/working-dir/logs/mirroring_errors_20240904_103019.txt for a list of mirroring errors.
 You may consider:
 * removing images or operators that cause the error from the image set config, and retrying
 * keeping the image set config (images are mandatory for you), and retrying
 * mirroring the failing images manually, if retries also fail.
2024/09/04 10:30:19  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

Scan the logs for any errors, including the file beginning with mirroring_errors, taking a copy of all files in /data/oc-mirror/workdir/working-dir/cluster-resources/ as you will need them for performing a disconnected install. Then create both imageContentSources and catalog sources in openshift-marketplace namespace after install.

In your cluster-resources folder, look for openshift-release-dev which you should find in idms-oc-mirror.yaml. This information is needed to perform the disconnected install. Ensure you add the lines from imageDigestMirrors to your install-config.yaml to be able to pull images from your local registry for Red Hat OpenShift installation:

  imageDigestMirrors:
  - mirrors:
    - registry.local.momolab.io:8443/mirror/openshift-release-dev
    source: quay.io/openshift-release-dev

Or:

  imageTagMirrors:
  - mirrors:
    - registry.local.momolab.io:8443/mirror/openshift-release-dev
    source: quay.io/openshift-release-dev

In install-config.yaml, add:

imageContentSources:
  - mirrors:
    - registry.local.momolab.io:8443/mirror/openshift-release-dev
    source: quay.io/openshift-release-dev

This will look for the relevant OpenShift install containers from your local registry. If you can’t find quay.io/openshift-release-dev in your files, it is most likely you encountered errors during the mirroring process.

Install disconnected single node OpenShift cluster from local Quay registry

Prior to installing disconnected OpenShift, ensure your install-config.yaml file contains the correct additionalTrustBundle and imageContentSources and that the registry is populated correctly (check oc-mirror logs).

This guide uses a single node OpenShift installation, as per these docs.

Once your installation begins, and the Red Hat Enterprise Linux CoreOS node becomes available, you will be able to check the installation status by logging in to your CoreOS node:

$ ssh -i yourkey.pem core@sno1.local.momolab.io

 And issuing the following command (this is also displayed when you log in, so it is easy to remember):

$ journalctl -b -f -u release-image.service -u bootkube.service

If installation succeeds, you should see:

Sep 09 03:30:10 sno1.local.momolab.io bootkube.sh[2074]: bootkube.service complete
Sep 09 03:30:10 sno1.local.momolab.io systemd[1]: bootkube.service: Deactivated successfully.
Sep 09 03:30:10 sno1.local.momolab.io systemd[1]: bootkube.service: Consumed 1min 57.667s CPU time.
Broadcast message from root@localhost (Sat 2024-09-14 03:30:59 UTC):
Bootstrap completed, server is going to reboot.
The system will reboot at Sat 2024-09-14 03:31:59 UTC!

There may be many reasons why an installation will fail, but a reason that could arise from either the images not being in the registry or by not applying the correct imageContentSources may look like this:

Sep 09 10:41:54 sno2.local.momolab.io bootkube.sh[251315]: Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf: (Mirrors also failed: [registry.local.momolab.io:8443/mirror/release/ocp-v4.0-art-dev@sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf: reading manifest sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf in registry.local.momolab.io:8443/mirror/release/ocp-v4.0-art-dev: name unknown: repository not found]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf: reading manifest sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized
Sep 09 10:41:54 sno2.local.momolab.io systemd[1]: bootkube.service: Main process exited, code=exited, status=125/n/a
Sep 09 10:41:54 sno2.local.momolab.io systemd[1]: bootkube.service: Failed with result 'exit-code'.
Sep 09 10:41:54 sno2.local.momolab.io systemd[1]: bootkube.service: Consumed 8.444s CPU time.

In this case, we notice:

reading manifest sha256:fef32434645e88bc014e1435ab571ae5336e1eadcc28d46eb7974e0348d3bfbf in registry.local.momolab.io:8443/mirror/release/ocp-v4.0-art-dev: name unknown: repository not found]): 

The correct mapping should be:

registry.local.momolab.io:8443/mirror/openshift-release-dev/ocp-v4.0-art-dev

Instead of:

registry.local.momolab.io:8443/mirror/release

Post install configuration

Once your cluster is installed successfully, there are some post-install tasks that assist in allowing your cluster to function in a disconnected environment.

From your oc-mirror command, you should have the following files:

• idms-oc-mirror.yaml: This is a list of mappings between the original public registry and your local registry for all images that are identified by their digest (ImageDigestMirrorSet).
• itms-oc-mirror.yaml: This is a list of mappings between the original public registry and your local registry for all images that are identified by their tag (ImageTagMirrorSet).
• cs-redhat-operator-index-v4-16.yaml: This is the CatalogSource for RedHat Operators. 

• cs-certified-operator-index-v4-16.yaml: This is the catalog for operators certified by Red Hat but not necessarily developed by Red Hat.
• cs-community-operator-index-v4-16.yaml: This is the catalog source for all community based operators.

Before importing these catalog sources, it is necessary to disable all default catalog sources with this command (as they will not work in a disconnected environment):

$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]

Then apply all of the files generated by oc-mirror, assuming they are all in a subdirectory called disconnected:

$ oc apply -f disconnected/
catalogsource.operators.coreos.com/cs-certified-operator-index-v4-16 created
catalogsource.operators.coreos.com/cs-community-operator-index-v4-16 created
catalogsource.operators.coreos.com/cs-redhat-operator-index-v4-16 created
catalogsource.operators.coreos.com/redhat-operators created
imagedigestmirrorset.config.openshift.io/idms-operator-0 configured
imagetagmirrorset.config.openshift.io/itms-operator-0 configured
imagetagmirrorset.config.openshift.io/itms-generic-0 configured

You should see something like this (redhat-operators must be there though):

$ oc get catalogsource -A
NAMESPACE               NAME                                DISPLAY   TYPE   PUBLISHER   AGE
openshift-marketplace   cs-certified-operator-index-v4-16             grpc               9d
openshift-marketplace   cs-community-operator-index-v4-16             grpc               9d
openshift-marketplace   redhat-operators                              grpc               9d

Conclusion

In this article, we demonstrated how to use oc-mirror v2 to install a disconnected Red Hat Single Node OpenShift cluster, and demonstrated how OpenShift operators are made available in a disconnected environment via the files generated by oc-mirror v2.

Although still in technology preview, oc-mirror v2 comes with several improvements on performance and resilience, which aim to make the disconnected mirroring experience a lot easier. Therefore, while still in technology preview, it is worth familiarizing oneself with this new version.

This article was only possible with the help of the oc-mirror team.