Upgrade Project Quay

Upgrade overview

The upgrade procedure for Project Quay depends on the type of installation you are using.

The Project Quay Operator provides a simple method to deploy and manage a Project Quay cluster. This is the preferred procedure for deploying Project Quay on OpenShift.

The Project Quay Operator should be upgraded using the Operator Lifecycle Manager (OLM) as described in the section "Upgrading Quay using the Quay Operator".

The procedure for upgrading a proof-of-concept or highly available installation of Project Quay and Clair is documented in the section "Standalone upgrade".

Upgrading the Quay Operator Overview

The Quay Operator follows a synchronized versioning scheme, which means that each version of the Operator is tied to the version of Quay and the components that it manages. There is no field on the QuayRegistry custom resource which sets the version of Quay to deploy; the Operator only knows how to deploy a single version of all components. This scheme was chosen to ensure that all components work well together and to reduce the complexity of the Operator needing to know how to manage the lifecycles of many different versions of Quay on Kubernetes.

Operator Lifecycle Manager

The Quay Operator should be installed and upgraded using the Operator Lifecycle Manager (OLM). When creating a Subscription with the default approvalStrategy: Automatic, OLM will automatically upgrade the Quay Operator whenever a new version becomes available.

Warning

When the Quay Operator is installed via Operator Lifecycle Manager, it may be configured to support automatic or manual upgrades. This option is shown on the Operator Hub page for the Quay Operator during installation. It can also be found in the Quay Operator Subscription object via the approvalStrategy field. Choosing Automatic means that your Quay Operator will automatically be upgraded whenever a new Operator version is released. If this is not desirable, then the Manual approval strategy should be selected.

Upgrading the Quay Operator

The standard approach for upgrading installed Operators on OpenShift is documented at Upgrading installed Operators.

In general, Project Quay supports upgrades from a prior (N-1) minor version only. For example, upgrading directly from Project Quay 3.0.5 to the latest version of 3.5 is not supported. Instead, users would have to upgrade as follows:

3.0.5 → 3.1.3
3.1.3 → 3.2.2
3.2.2 → 3.3.4
3.3.4 → 3.4.z
3.4.z → 3.5.z

This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.

In some cases, Project Quay supports direct, single-step upgrades from prior (N-2, N-3) minor versions. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases. The following upgrade paths are supported:

3.3.z → 3.6.z
3.4.z → 3.6.z
3.4.z → 3.7.z
3.5.z → 3.7.z

For users on standalone deployments of Quay wanting to upgrade to 3.7, see the Standalone upgrade guide.

Upgrading Quay

To update Quay from one minor version to the next, for example, 3.4 → 3.5, you need to change the update channel for the Quay Operator.

For z stream upgrades, for example, 3.4.2 → 3.4.3, updates are released in the major-minor channel that the user initially selected during install. The procedure to perform a z stream upgrade depends on the approvalStrategy as outlined above. If the approval strategy is set to Automatic, the Quay Operator will upgrade automatically to the newest z stream. This results in automatic, rolling Quay updates to newer z streams with little to no downtime. Otherwise, the update must be manually approved before installation can begin.

Notes on upgrading directly from 3.3.z or 3.4.z to 3.6

Upgrading with edge routing enabled

Previously, when running a 3.3.z version of Project Quay with edge routing enabled, users were unable to upgrade to 3.4.z versions of Project Quay. This has been resolved with the release of Project Quay 3.6.

When upgrading from 3.3.z to 3.6, if tls.termination is set to none in your Project Quay 3.3.z deployment, it will change to HTTPS with TLS edge termination and use the default cluster wildcard certificate. For example:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quay33
spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    enableRepoMirroring: true
    image: quay.io/quay/quay:v3.3.4-2
    ...
    externalAccess:
      hostname: quayv33.apps.devcluster.openshift.com
      tls:
        termination: none
    database:
...

Upgrading with custom TLS certificate/key pairs without Subject Alternative Names

There is an issue for customers using their own TLS certificate/key pairs without Subject Alternative Names (SANs) when upgrading from Project Quay 3.3.4 to Project Quay 3.6 directly. During the upgrade to Project Quay 3.6, the deployment is blocked, with the error message from the Quay Operator pod logs indicating that the Quay TLS certificate must have SANs.

If possible, you should regenerate your TLS certificates with the correct hostname in the SANs. A possible workaround involves defining an environment variable in the quay-app, quay-upgrade and quay-config-editor pods after upgrade to enable CommonName matching:

 GODEBUG=x509ignoreCN=0

The GODEBUG=x509ignoreCN=0 flag enables the legacy behavior of treating the CommonName field on X.509 certificates as a host name when no SANs are present. However, this workaround is not recommended, as it will not persist across a redeployment.

Configuring Clair v4 when upgrading from 3.3.z or 3.4.z to 3.6 using the Quay Operator

To set up Clair v4 on a new Project Quay deployment on OpenShift, it is highly recommended to use the Quay Operator. By default, the Quay Operator will install or upgrade a Clair deployment along with your Project Quay deployment and configure Clair security scanning automatically.

For instructions on setting up Clair v4 on OpenShift, see Setting Up Clair on a Project Quay OpenShift deployment.

Swift configuration when upgrading from 3.3.z to 3.6

When upgrading from Project Quay 3.3.z to 3.6.z, some users might receive the following error: Switch auth v3 requires tenant_id (string) in os_options. As a workaround, you can manually update your DISTRIBUTED_STORAGE_CONFIG to add the os_options and tenant_id parameters:

  DISTRIBUTED_STORAGE_CONFIG:
    brscale:
    - SwiftStorage
    - auth_url: http://****/v3
      auth_version: "3"
      os_options:
        tenant_id: ****
        project_name: ocp-base
        user_domain_name: Default
      storage_path: /datastorage/registry
      swift_container: ocp-svc-quay-ha
      swift_password: *****
      swift_user: *****

Changing the update channel for an Operator

The subscription of an installed Operator specifies an update channel, which is used to track and receive updates for the Operator. To upgrade the Quay Operator to start tracking and receiving updates from a newer channel, change the update channel in the Subscription tab for the installed Quay Operator. For subscriptions with an Automatic approval strategy, the upgrade begins automatically and can be monitored on the page that lists the Installed Operators.

Manually approving a pending Operator upgrade

If an installed Operator has the approval strategy in its subscription set to Manual, when new updates are released in its current update channel, the update must be manually approved before installation can begin. If the Quay Operator has a pending upgrade, this status will be displayed in the list of Installed Operators. In the Subscription tab for the Quay Operator, you can preview the install plan and review the resources that are listed as available for upgrade. If satisfied, click Approve and return to the page that lists Installed Operators to monitor the progress of the upgrade.

The following image shows the Subscription tab in the UI, including the update Channel, the Approval strategy, the Upgrade status and the InstallPlan:

Subscription tab including upgrade Channel and Approval strategy

The list of Installed Operators provides a high-level summary of the current Quay installation:

Installed Operators

Upgrading a QuayRegistry

When the Quay Operator starts, it immediately looks for any QuayRegistries it can find in the namespace(s) it is configured to watch. When it finds one, the following logic is used:

If status.currentVersion is unset, reconcile as normal.
If status.currentVersion equals the Operator version, reconcile as normal.
If status.currentVersion does not equal the Operator version, check if it can be upgraded. If it can, perform upgrade tasks and set the status.currentVersion to the Operator’s version once complete. If it cannot be upgraded, return an error and leave the QuayRegistry and its deployed Kubernetes objects alone.

Enabling features in Quay 3.7

Quota management configuration

Quota management is now supported under the FEATURE_QUOTA_MANAGEMENT property and is turned off by default. To enable quota management, set the feature flag in your config.yaml to true:

FEATURE_QUOTA_MANAGEMENT: true

Using Project Quay to proxy a remote organization configuration

Using Project Quay to proxy a remote organization is now supported under the FEATURE_PROXY_CACHE property. To enable proxy cache, set the feature flag in your confg.yaml to true:

FEATURE_PROXY_CACHE: true

Project Quay build enhancements

Builds can be run on virtualized platforms. Backwards compatibility to run previous build configurations are also available. To enable virtual builds, set the feature flag in your config.yaml to true:

FEATURE_BUILD_SUPPORT: true

Geo-replication using the Project Quay Operator

Deployments of Project Quay with geo-replication is now supported by Operator deployments. To enable geo-replication, set the feature flag in your config.yaml to true:

FEATURE_STORAGE_REPLICATION: true

Enabling features in Quay 3.6

Console monitoring and alerting

The support for monitoring Quay 3.6 in the OpenShift console requires that the Operator is installed in all namespaces. If you previously installed the Operator in a specific namespace, delete the Operator itself and reinstall it for all namespaces once the upgrade has taken place.

OCI and Helm support

Support for Helm and some OCI artifacts is now enabled by default in Project Quay 3.6. If you want to explicitly enable the feature, for example, if you are upgrading from a version where it is not enabled by default, you need to reconfigure your Quay deployment to enable the use of OCI artifacts using the following properties:

FEATURE_GENERAL_OCI_SUPPORT: true

Upgrading a QuayEcosystem

Upgrades are supported from previous versions of the Operator which used the QuayEcosystem API for a limited set of configurations. To ensure that migrations do not happen unexpectedly, a special label needs to be applied to the QuayEcosystem for it to be migrated. A new QuayRegistry will be created for the Operator to manage, but the old QuayEcosystem will remain until manually deleted to ensure that you can roll back and still access Quay in case anything goes wrong. To migrate an existing QuayEcosystem to a new QuayRegistry, follow these steps:

Add "quay-operator/migrate": "true" to the metadata.labels of the QuayEcosystem.

$ oc edit quayecosystem <quayecosystemname>

metadata:
  labels:
    quay-operator/migrate: "true"

Wait for a QuayRegistry to be created with the same metadata.name as your QuayEcosystem. The QuayEcosystem will be marked with the label "quay-operator/migration-complete": "true".
Once the status.registryEndpoint of the new QuayRegistry is set, access Quay and confirm all data and settings were migrated successfully.
When you are confident everything worked correctly, you may delete the QuayEcosystem and Kubernetes garbage collection will clean up all old resources.

Reverting QuayEcosystem Upgrade

If something goes wrong during the automatic upgrade from QuayEcosystem to QuayRegistry, follow these steps to revert back to using the QuayEcosystem:

Delete the QuayRegistry using either the UI or kubectl:

$ kubectl delete -n <namespace> quayregistry <quayecosystem-name>

If external access was provided using a Route, change the Route to point back to the original Service using the UI or kubectl.

Note

If your QuayEcosystem was managing the Postgres database, the upgrade process will migrate your data to a new Postgres database managed by the upgraded Operator. Your old database will not be changed or removed but Quay will no longer use it once the migration is complete. If there are issues during the data migration, the upgrade process will exit and it is recommended that you continue with your database as an unmanaged component.

Supported QuayEcosystem Configurations for Upgrades

The Quay Operator will report errors in its logs and in status.conditions if migrating a QuayEcosystem component fails or is unsupported. All unmanaged components should migrate successfully because no Kubernetes resources need to be adopted and all the necessary values are already provided in Quay’s config.yaml.

Database

Ephemeral database not supported (volumeSize field must be set).

Redis

Nothing special needed.

External Access

Only passthrough Route access is supported for automatic migration. Manual migration required for other methods.

LoadBalancer without custom hostname: After the QuayEcosystem is marked with label "quay-operator/migration-complete": "true", delete the metadata.ownerReferences field from existing Service before deleting the QuayEcosystem to prevent Kubernetes from garbage collecting the Service and removing the load balancer. A new Service will be created with metadata.name format <QuayEcosystem-name>-quay-app. Edit the spec.selector of the existing Service to match the spec.selector of the new Service so traffic to the old load balancer endpoint will now be directed to the new pods. You are now responsible for the old Service; the Quay Operator will not manage it.
LoadBalancer/NodePort/Ingress with custom hostname: A new Service of type LoadBalancer will be created with metadata.name format <QuayEcosystem-name>-quay-app. Change your DNS settings to point to the status.loadBalancer endpoint provided by the new Service.

Clair

Nothing special needed.

Object Storage

QuayEcosystem did not have a managed object storage component, so object storage will always be marked as unmanaged. Local storage is not supported.

Repository Mirroring

Nothing special needed.

Standalone upgrade

3.0.5 → 3.1.3
3.1.3 → 3.2.2
3.2.2 → 3.3.4
3.3.4 → 3.4.z
3.4.z → 3.5.z

This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.

3.3.z → 3.6.z
3.4.z → 3.6.z
3.4.z → 3.7.z
3.5.z → 3.7.z

For users wanting to upgrade via the Quay Operator, see Upgrading Quay by upgrading the Quay Operator.

This document describes the steps needed to perform each individual upgrade. Determine your current version and then follow the steps in sequential order, starting with your current version and working up to your desired target version.

See the Project Quay Release Notes for information on features for individual releases.

The general procedure for a manual upgrade consists of the following steps:

Stop the Quay and Clair containers.
Backup the database and image storage (optional but recommended).
Start Clair using the new version of the image.
Wait until Clair is ready to accept connections before starting the new version of Quay.

Accessing images

Images for Quay 3.4.0 and later are available from registry.redhat.io and registry.access.redhat.com, with authentication set up as described in Red Hat Container Registry Authentication.

Images for Quay 3.3.4 and earlier are available from quay.io, with authentication set up as described in Accessing Project Quay without a CoreOS login.

Upgrade to 3.7.z from 3.6.z

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade to 3.7.z from 3.5.z

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade to 3.7.z from 3.4.z

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade to 3.7.z from 3.3.z

Upgrading to Project Quay 3.7 from 3.3. is unsupported. Users must first upgrade to 3.6 from 3.3, and then upgrade to 3.7. For more information, see Upgrade to 3.6.z from 3.3.z.

Upgrade to 3.6.z from 3.5.z

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade to 3.6.z from 3.4.z

Note	Project Quay 3.6 supports direct, single-step upgrade from 3.4.z. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases.

Upgrading to Project Quay 3.6 from 3.4.z requires a database migration which does not support downgrading back to a prior version of Project Quay. Please back up your database before performing this migration.

Users will also need to configure a completely new Clair v4 instance to replace the old Clair v2 when upgrading from 3.4.z. For instructions on configuring Clair v4, see Setting up Clair on a non-OpenShift Project Quay deployment.

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade to 3.6.z from 3.3.z

Note	Project Quay 3.6 supports direct, single-step upgrade from 3.3.z. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases.

Upgrading to Project Quay 3.6.z from 3.3.z requires a database migration which does not support downgrading back to a prior version of Project Quay. Please back up your database before performing this migration.

Users will also need to configure a completely new Clair v4 instance to replace the old Clair v2 when upgrading from 3.3.z. For instructions on configuring Clair v4, see Setting up Clair on a non-OpenShift Project Quay deployment.

Target images

Quay: quay.io/projectquay/quay:v3.7.8
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Swift configuration when upgrading from 3.3.z to 3.6

  DISTRIBUTED_STORAGE_CONFIG:
    brscale:
    - SwiftStorage
    - auth_url: http://****/v3
      auth_version: "3"
      os_options:
        tenant_id: ****
        project_name: ocp-base
        user_domain_name: Default
      storage_path: /datastorage/registry
      swift_container: ocp-svc-quay-ha
      swift_password: *****
      swift_user: *****

Upgrade to 3.5.7 from 3.4.z

Target images

Quay: quay.io/projectquay/quay:v3.5.1
Clair: quay.io/projectquay/clair:4.4.4
PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
Redis: registry.redhat.io/rhel8/redis-5:1

Upgrade Quay Bridge Operator

To upgrade the Quay Bridge Operator (QBO), change the Channel Subscription update channel in the Subscription tab to the desired channel.

When upgrading QBO from version 3.5 to 3.7, a number of extra steps are required:

You need to create a new QuayIntegration custom resource. This can be completed in the Web Console or from the command line.

upgrade-quay-integration.yaml

- apiVersion: quay.redhat.com/v1
  kind: QuayIntegration
  metadata:
    name: example-quayintegration-new
  spec:
    clusterID: openshift (1)
    credentialsSecret:
      name: quay-integration
      namespace: openshift-operators
    insecureRegistry: false
    quayHostname: https://registry-quay-quay35.router-default.apps.cluster.openshift.com

Make sure that the clusterID matches the value for the existing QuayIntegration resource.

Create the new QuayIntegration custom resource:
```
$ oc create -f upgrade-quay-integration.yaml
```
Delete the old QuayIntegration custom resource.

Delete the old mutatingwebhookconfigurations:

$ oc delete mutatingwebhookconfigurations.admissionregistration.k8s.io quay-bridge-operator