Deploy Project Quay on OpenShift with the Quay Operator

You need a privileged account to an OpenShift 4.5 or later cluster on which to deploy the Project Quay Operator. That account must have the ability to create namespaces at the cluster scope.

Each Project Quay application pod has the following resource requirements:

The Project Quay Operator will create at least one application pod per Project Quay deployment it manages. Ensure your OpenShift cluster has sufficient compute resources for these requirements.

By default, the Project Quay Operator uses the ObjectBucketClaim Kubernetes API to provision object storage. Consuming this API decouples the Operator from any vendor-specific implementation. OpenShift Container Storage provides this API via its NooBaa component, which will be used in this example.

Project Quay can be manually configured to use any of the following supported cloud storage options:

Set the config option FEATURE_USER_INITIALIZE to true, so that you can use the API /api/v1/user/initialize to create the first user. This API endpoint does not require authentication, unlike all other registry API calls which require an OAuth token which is generated by an OAuth application in an existing organization.

Once you have deployed Quay, you can use the API to create a user, for example, quayadmin, provided no other users have already been created. For more information, see the section on Creating the first user using the API

Set the config option BROWSER_API_CALLS_XHR_ONLY to false, to allow general access to the Quay registry API.

While you cannot create a user until after deployment, it is convenient to ensure that first user is an administrator with full permissions. It is easier to configure this in advance, using the SUPER_USER configuration object.

Once you have configured a super user, you can restrict the ability to create new users to the super user group. Set the FEATURE_USER_CREATION to false to restrict user creation.

Create a config.yaml configuration file that includes the appropriate settings:

Some configuration examples for unmanaged storage are provided in this section for convenience. See the Project Quay configuration guide for full details for setting up object storage.

If you want the Operator to manage object storage for Quay, your cluster needs to be capable of providing it via the ObjectBucketClaim API. There are multiple implementations of this API available, for instance, NooBaa in combination with Kubernetes PersistentVolumes or scalable storage backends like Ceph. Refer to the NooBaa documentation for more details on how to deploy this component.

You configure the connection to the database using the required DB_URI field and optional connection arguments in the DB_CONNECTION_ARGS structure. Some key-value pairs defined under DB_CONNECTION_ARGS are generic while others are database-specific. In particular, SSL configuration depends on the database you are deploying, and examples for PostgreSQL and MySQL are given below.

Recommendations:

To add your own TLS cert and key, include them in the config bundle secret as follows:

If you wish to use an external Redis database, set the component as unmanaged in the QuayRegistry instance:

HorizontalPodAutoscalers have been added to the Clair, Quay, and Mirror pods, so that they now automatically scale during load spikes.

As HPA is configured by default to be managed, the number of pods for Quay, Clair and repository mirroring is set to two. This facilitates the avoidance of downtime when updating / reconfiguring Quay via the Operator or during rescheduling events.

If you wish to disable autoscaling or create your own HorizontalPodAutoscaler, simply specify the component as unmanaged in the QuayRegistry instance:

To prevent the Operator from creating a Route:

If you install the Quay Operator in a single namespace, the monitoring component is automatically set to 'unmanaged'. To enable monitoring in this scenario, see the section Enabling monitoring when Operator is installed in a single namespace.

To disable monitoring explicitly:

To disable mirroring explicitly:

Use the oc get pods command to view the deployed components:

A default deployment shows the following running pods:

As HPA is configured by default to be managed, the number of pods for Quay, Clair and repository mirroring is set to two. This facilitates the avoidance of downtime when updating / reconfiguring Quay via the Operator or during rescheduling events.

When using the API to create the first user, the following conditions must be met:

For more information on pre-configuring the deployment, see the section Pre-configuring Quay for automation

Project Quay 3.6 provides new functionality to troubleshoot problems during the deployment phase. The status in the QuayRegistry object can help you monitor the health of the components during the deployment an help you debug any problems that may arise:

Immediately after deployment, the QuayRegistry object will show the basic configuration:

Use the oc get pods command to view the current state of the deployed components:

While the deployment is in progress, the QuayRegistry object will show the current status. In this instance, database migrations are taking place, and other components are waiting until this completes.

When the deployment process finishes successfully, the status in the QuayRegistry object shows no unhealthy components:

Browse to the config editor endpoint and then enter the username, typically quayconfig, and the corresponding password to access the config tool:

In this example of updating the configuration, a superuser is added via the config editor tool:

After reconfiguring the Operator, you can track the progress of the redeployment in the YAML tab for the specific instance of QuayRegistry, in this case, example-registry:

Each time the status changes, you will be prompted to reload the data to see the updated version. Eventually, the Operator will reconcile the changes, and there will be no unhealthy components reported.

The Events tab for the QuayRegistry shows some events related to the redeployment:

Streaming events, for all resources in the namespace that are affected by the reconfiguration, are available in the OpenShift console under Home → Events:

Since a new pod has been created for the config tool, a new secret will have been created, and you will need to use the updated password when you next attempt to login:

Use the config bundle to access the updated config.yaml file.

In the OpenShift console, navigate to Monitoring → Dashboards and search for the dashboard of your desired Quay registry instance:

The dashboard shows various statistics including:

You can see the underlying metrics behind the Quay dashboard, by accessing Monitoring → Metrics in the UI. In the Expression field, enter the text quay_ to see the list of metrics available:

Select a sample metric, for example, quay_org_rows:

This metric shows the number of organizations in the registry, and it is directly surfaced in the dashboard as well.

An alert is raised if the Quay pods restart too often. The alert can be configured by accessing the Alerting rules tab from Monitoring → Alerting in the consol UI and searching for the Quay-specific alert:

Select the QuayPodFrequentlyRestarting rule detail to configure the alert:

To obtain the clairctl program from a Clair deployment in an OpenShift cluster, use the oc cp command, for example:

For a standalone Clair deployment, use the podman cp command, for example:

From a Clair instance that has access to the internet, use clairctl with the appropriate configuration file to export the updaters bundle:

After transferring the updaters bundle to the air-gapped environment, use clairctl to import the bundle into the Clair database deployed by the OpenShift Operator:

In the cluster used in this example, there are three master nodes and six worker nodes:

Label the final three worker nodes for infrastructure use:

Now, when you list the nodes in the cluster, the last 3 worker nodes will have an added role of infra:

With an infra node being assigned as a worker, there is a chance that user workloads could get inadvertently assigned to an infra node. To avoid this, you can apply a taint to the infra node and then add tolerations for the pods you want to control.

If you have already deployed Quay using the Quay Operator, remove the installed operator and any specific namespace(s) you created for the deployment.

Create a Project resource, specifying a node selector and toleration as shown in the following example:

Use the oc apply command to create the project:

Any subsequent resources created in the quay-registry namespace should now be scheduled on the dedicated infrastructure nodes.

When installing the Quay Operator, specify the appropriate project namespace explicitly, in this case quay-registry. This will result in the operator pod itself landing on one of the three infrastructure nodes:

Create the registry as explained earlier, and then wait for the deployment to be ready. When you list the Quay pods, you should now see that they have only been scheduled on the three nodes that you have labelled for infrastructure purposes:

Configure OpenShift Monitoring to scrape the metrics by creating a ServiceMonitor resource.

You can access the metrics in the OpenShift console under Monitoring → Metrics. In the Expression field, enter the text quay_ to see the list of metrics available:

For example, if you have added users to your registry, select the quay-users_rows metric:

After a few minutes (depending on the size of the PVC), the expanded size should reflect in the PVC’s Capacity field.

After a few minutes, the additional storage pool will be added to the Noobaa resources and available for use by Project Quay.

The following environment variables are used in the Operator to override component images:

When the Quay Operator is installed in a cluster via the Operator Lifecycle Manager (OLM), the managed component container images can be easily overridden by modifying the ClusterServiceVersion object, which is OLM’s representation of a running Operator in the cluster. Find the Quay Operator’s ClusterServiceVersion either by using a Kubernetes UI or kubectl/oc:

Using the UI, oc edit, or any other method, modify the Quay ClusterServiceVersion to include the environment variables outlined above to point to the override images:

JSONPath: spec.install.spec.deployments[0].spec.template.spec.containers[0].env

Note that this is done at the Operator level, so every QuayRegistry will be deployed using these same overrides.

From a Project Quay point of view, to update from one minor version to the next, for example, 3.4 → 3.5, you need to actively change the update channel for the Quay Operator.

For z stream upgrades, for example, 3.4.2 → 3.4.3, updates are released in the major-minor channel that the user initially selected during install. The procedure to perform a z stream upgrade depends on the approvalStrategy as outlined above. If the approval strategy is set to Automatic, the Operator will upgrade automatically to the newest z stream, resulting in automatic, rolling Quay updates to newer z streams with little to no downtime. Otherwise, the update must be manually approved before installation can begin.

The subscription of an installed Operator specifies an update channel, which is used to track and receive updates for the Operator. To upgrade the Quay Operator to start tracking and receiving updates from a newer channel, change the update channel in the Subscription tab for the installed Quay Operator. For subscriptions with an Automatic approval strategy, the upgrade begins automatically and can be monitored on the page that lists the Installed Operators.

If an installed Operator has the approval strategy in its subscription set to Manual, when new updates are released in its current update channel, the update must be manually approved before installation can begin. If the Quay Operator has a pending upgrade, this status will be displayed in the list of Installed Operators. In the Subscription tab for the Quay Operator, you can preview the install plan and review the resources that are listed as available for upgrade. If satisfied, click Approve and return to the page that lists Installed Operators to monitor the progress of the upgrade.

The following image shows the Subscription tab in the UI, including the update Channel, the Approval strategy, the Upgrade status and the InstallPlan:

The list of Installed Operators provides a high-level summary of the current Quay installation:

The support for monitoring of Quay 3.6 in the OpenShift console requires that the Operator is installed in all namespaces. If you previously installed the Operator in a specific namespace, delete the Operator itself and reinstall it for all namespaces, once the upgrade has taken place.

Support for Helm and some OCI artifacts is now enabled by default in Project Quay {producty}. If you want to explicitly enable the feature, for example, if you are upgrading from a version where it is not enabled by default, you need to reconfigure your Quay deployment to enable the use of OCI artifacts using the following properties:

If something goes wrong during the automatic upgrade from QuayEcosystem to QuayRegistry, follow these steps to revert back to using the QuayEcosystem:

The Quay Operator will report errors in its logs and in status.conditions if migrating a QuayEcosystem component fails or is unsupported. All unmanaged components should migrate successfully because no Kubernetes resources need to be adopted and all the necessary values are already provided in Quay’s config.yaml.

Database

Ephemeral database not supported (volumeSize field must be set).

Redis

Nothing special needed.

External Access

Only passthrough Route access is supported for automatic migration. Manual migration required for other methods.

Clair

Nothing special needed.

Object Storage

QuayEcosystem did not have a managed object storage component, so object storage will always be marked as unmanaged. Local storage is not supported.

Repository Mirroring

Nothing special needed.