To configure and manage advanced capabilities in Red Hat Quay on OpenShift Container Platform, you can use this guide. It covers features such as vulnerability scanning with Clair, geo-replication, backup and restore operations, FIPS compliance, monitoring and alerting, and custom SSL certificates.

Federal Information Processing Standard (FIPS) readiness and compliance

The Federal Information Processing Standard (FIPS) developed by the National Institute of Standards and Technology (NIST) is regarded as the highly regarded for securing and encrypting sensitive data, notably in highly regulated areas such as banking, healthcare, and the public sector. Red Hat Enterprise Linux (RHEL) and OpenShift Container Platform support FIPS by providing a FIPS mode, in which the system only allows usage of specific FIPS-validated cryptographic modules like openssl. This ensures FIPS compliance.

Enable FIPS compliance

To enable FIPS compliance for your Project Quay deployment, you can set the FEATURE_FIPS configuration field to True in your config.yaml file. This ensures that Project Quay uses only FIPS-validated cryptographic modules for securing sensitive data.

Prerequisite
  • If you are running a standalone deployment of Project Quay, your Red Hat Enterprise Linux (RHEL) deployment is version 8 or later and FIPS-enabled.

  • If you are deploying Red Hat Quay on OpenShift Container Platform, OpenShift Container Platform is version 4.10 or later.

  • Your Project Quay version is 3.5.0 or later.

  • If you are using the Red Hat Quay on OpenShift Container Platform on an IBM Power or IBM Z cluster:

    • OpenShift Container Platform version 4.14 or later is required

    • Project Quay version 3.10 or later is required

  • You have administrative privileges for your Project Quay deployment.

Procedure
  • In your Project Quay config.yaml file, set the FEATURE_FIPS configuration field to True. For example:

    # ...
    FEATURE_FIPS = true
    # ...

    With FEATURE_FIPS set to True, Project Quay runs using FIPS-compliant hash functions.

Console monitoring and alerting

Project Quay provides monitoring and alerting features in the OpenShift Container Platform console for instances deployed by the Operator. You can use Grafana dashboards, individual metrics, and alerts to monitor registry performance and receive notifications when Quay pods restart frequently.

Note

To enable the monitoring features, you must select All namespaces on the cluster as the installation mode when installing the Project Quay Operator.

Dashboard

On the OpenShift Container Platform console, click Monitoring → Dashboards and search for the dashboard of your desired Project Quay registry instance:

Choose Quay dashboard

The dashboard shows various statistics including the following:

  • The number of Organizations, Repositories, Users, and Robot accounts

  • CPU Usage

  • Max memory usage

  • Rates of pulls and pushes, and authentication requests

  • API request rate

  • Latencies

Console dashboard

Metrics

You can see the underlying metrics behind the Project Quay dashboard by accessing Monitoring → Metrics in the UI. In the Expression field, enter the text quay_ to see the list of metrics available:

Quay metrics

Select a sample metric, for example, quay_org_rows:

Number of Quay organizations

This metric shows the number of organizations in the registry. It is also directly surfaced in the dashboard.

Alerting

An alert is raised if the Quay pods restart too often. The alert can be configured by accessing the Alerting rules tab from Monitoring → Alerting in the console UI and searching for the Quay-specific alert:

Alerting rules

Select the QuayPodFrequentlyRestarting rule detail to configure the alert:

Alerting rule details

Clair security scanner

Clair is a vulnerability scanner that analyzes container image layers and reports security issues from upstream CVE databases in Project Quay. You can use these reports to identify vulnerabilities that affect image content.

Clair v4 (Clair) leverages static code analyses for parsing image content and reporting vulnerabilities affecting the content. Quay.io includes Clair, enables it automatically, and the Project Quay development team manages it.

For Quay.io users, images are automatically indexed after they are pushed to your repository. Reports are then fetched from Clair, which matches images against its CVE database to report security information. This process happens automatically on Quay.io, and manual rescans are not required.

For standalone or Operator-based deployments of Project Quay, Clair can be deployed and scaled separately. Configuration fields for Clair are documented in the Clair configuration overview. Red Hat removes this chapter in a future version of Project Quay.

Setting up Clair on OpenShift Container Platform

The Project Quay Operator installs and configures Clair automatically when you deploy Project Quay on OpenShift Container Platform. You can increase Clair scratch storage for large images by configuring ephemeral volume overrides on the clair component.

Testing Clair

To confirm that Clair is scanning images on Project Quay, you can push a sample image to your registry and open the vulnerability report in the UI.

Prerequisites
  • You have deployed the Clair container image.

Procedure
  1. Pull a sample image by entering the following command:

    $ podman pull ubuntu:20.04
  2. Tag the image to your registry by entering the following command:

    $ sudo podman tag docker.io/library/ubuntu:20.04 <quay-server.example.com>/<user-name>/ubuntu:20.04
  3. Push the image to your Project Quay registry by entering the following command:

    $ sudo podman push --tls-verify=false quay-server.example.com/quayadmin/ubuntu:20.04
  4. Log in to your Project Quay deployment through the UI.

  5. Click the repository name, for example, quayadmin/ubuntu.

  6. In the navigation pane, click Tags.

    Security scan information appears for scanned repository images

  7. Click the image report, for example, 45 medium, to show a more detailed report:

    See all vulnerabilities or only those that are fixable

    Note

    In some cases, Clair shows duplicate reports on images, for example, ubi8/nodejs-12 or ubi8/nodejs-16. This occurs because vulnerabilities with the same name are for different packages. You can expect this behavior with Clair vulnerability reporting. Duplicate reports are not a bug.

Advanced Clair configuration

Advanced Clair configuration lets you customize Clair settings beyond the default installation. You can use these options to adjust scanning behavior, database connections, and other advanced features to meet specific security and performance requirements.

Unmanaged Clair configuration

Unmanaged Clair configuration lets you run a custom Clair deployment or use an external Clair database with the Project Quay Operator. You can use this option for geo-replicated environments or highly available databases outside your cluster.

Run a custom Clair configuration with an unmanaged Clair database

To use an external Clair database with the Project Quay Operator, you can set the clairpostgres component to unmanaged in your QuayRegistry custom resource.

Important

You must not use the same externally managed PostgreSQL database for both Project Quay and Clair deployments. Your PostgreSQL database must also not be shared with other workloads, as it might exhaust the natural connection limit on the PostgreSQL side when connection-intensive workloads, like Project Quay or Clair, contend for resources. Additionally, pgBouncer is not supported with Project Quay or Clair, so pgBouncer is not an option to resolve this issue.

Procedure
  • In the Quay Operator, set the clairpostgres component of the QuayRegistry custom resource to managed: false:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: quay370
    spec:
      configBundleSecret: config-bundle-secret
      components:
        - kind: objectstorage
          managed: false
        - kind: route
          managed: true
        - kind: tls
          managed: false
        - kind: clairpostgres
          managed: false
Configure a custom Clair database with an unmanaged Clair database

To configure a custom Clair database with SSL/TLS certificates on Project Quay, you can create a Quay configuration bundle secret that includes the clair-config.yaml file.

Note

The following procedure configures Clair with SSL/TLS certificates.

Procedure
  1. Create a Quay configuration bundle secret that includes the clair-config.yaml by entering the following command:

    $ oc create secret generic --from-file config.yaml=./config.yaml --from-file extra_ca_cert_rds-ca-2019-root.pem=./rds-ca-2019-root.pem --from-file clair-config.yaml=./clair-config.yaml --from-file ssl.cert=./ssl.cert --from-file ssl.key=./ssl.key config-bundle-secret
    Example Clair config.yaml file
    indexer:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslrootcert=/run/certs/rds-ca-2019-root.pem sslmode=verify-ca
        layer_scan_concurrency: 6
        migrations: true
        scanlock_retry: 11
    log_level: debug
    matcher:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslrootcert=/run/certs/rds-ca-2019-root.pem sslmode=verify-ca
        migrations: true
    metrics:
        name: prometheus
    notifier:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslrootcert=/run/certs/rds-ca-2019-root.pem sslmode=verify-ca
        migrations: true
    Note
    • The database certificate is mounted under /run/certs/rds-ca-2019-root.pem on the Clair application pod in the clair-config.yaml. It must be specified when configuring your clair-config.yaml.

  2. Add the clair-config.yaml file to your bundle secret, for example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: config-bundle-secret
      namespace: quay-enterprise
    data:
      config.yaml: <base64 encoded Quay config>
      clair-config.yaml: <base64 encoded Clair config>
      extra_ca_cert_<name>: <base64 encoded ca cert>
      ssl.crt: <base64 encoded SSL certificate>
      ssl.key: <base64 encoded SSL private key>
    Note

    When updated, the provided clair-config.yaml file is mounted into the Clair pod. Any fields not provided are automatically populated with defaults using the Clair configuration module.

  3. You can check the status of your Clair pod by clicking the commit in the Build History page, or by running oc get pods -n <namespace>. For example:

    $ oc get pods -n <namespace>
    Example output
    NAME                                               READY   STATUS    RESTARTS   AGE
    f192fe4a-c802-4275-bcce-d2031e635126-9l2b5-25lg2   1/1     Running   0          7s

Running a custom Clair configuration with a managed Clair database

You can customize Clair settings while the Project Quay Operator manages the Clair database. Use this approach to disable updater resources or configure Clair for disconnected environments.

Note
  • If you are running Project Quay in a disconnected environment, the airgap parameter of your clair-config.yaml must be set to True.

  • If you are running Project Quay in a disconnected environment, you should disable all updater components.

Set a Clair database to managed

To have the Project Quay Operator manage your Clair database, you can set the clairpostgres component to managed in your QuayRegistry custom resource.

Procedure
  • In the Quay Operator, set the clairpostgres component of the QuayRegistry custom resource to managed: true:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: quay370
    spec:
      configBundleSecret: config-bundle-secret
      components:
        - kind: objectstorage
          managed: false
        - kind: route
          managed: true
        - kind: tls
          managed: false
        - kind: clairpostgres
          managed: true
Configure a custom Clair database with a managed Clair configuration

To supply a custom clair-config.yaml while the Operator manages Clair on Project Quay, you can create a Quay configuration bundle secret that includes your Clair configuration file.

Procedure
  1. Create a Quay configuration bundle secret that includes the clair-config.yaml by entering the following command:

    $ oc create secret generic --from-file config.yaml=./config.yaml --from-file extra_ca_cert_rds-ca-2019-root.pem=./rds-ca-2019-root.pem --from-file clair-config.yaml=./clair-config.yaml config-bundle-secret
    Example Clair config.yaml file
    indexer:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslmode=disable
        layer_scan_concurrency: 6
        migrations: true
        scanlock_retry: 11
    log_level: debug
    matcher:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslmode=disable
        migrations: true
    metrics:
        name: prometheus
    notifier:
        connstring: host=quay-server.example.com port=5432 dbname=quay user=quayrdsdb password=quayrdsdb sslmode=disable
        migrations: true
    Note

    The database certificate is mounted under /run/certs/rds-ca-2019-root.pem on the Clair application pod in the clair-config.yaml. It must be specified when configuring your clair-config.yaml.

  2. Add the clair-config.yaml file to your bundle secret, for example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: config-bundle-secret
      namespace: quay-enterprise
    data:
      config.yaml: <base64 encoded Quay config>
      clair-config.yaml: <base64 encoded Clair config>
    Note

    When updated, the provided clair-config.yaml file is mounted into the Clair pod. Any fields not provided are automatically populated with defaults using the Clair configuration module.

  3. You can check the status of your Clair pod by clicking the commit in the Build History page, or by running oc get pods -n <namespace>. For example:

    $ oc get pods -n <namespace>
    Example output
    NAME                                               READY   STATUS    RESTARTS   AGE
    f192fe4a-c802-4275-bcce-d2031e635126-9l2b5-25lg2   1/1     Running   0          7s

Clair in disconnected environments

Clair supports disconnected Project Quay deployments that have no direct internet access. You can use the clairctl tool to transfer vulnerability database updates from an open host so Clair can scan images offline.

Clair uses a set of components called updaters to handle the fetching and parsing of data from various vulnerability databases. By default, updaters pull vulnerability data directly from the internet and work for immediate use.

Note

Currently, Clair enrichment data is CVSS data. Enrichment data is currently unsupported in disconnected environments.

Set up Clair in a disconnected OpenShift Container Platform cluster

To install the clairctl command-line utility on a disconnected OpenShift Container Platform cluster, you can copy the binary from a running Clair pod and make it executable.

Procedure
  1. Install the clairctl program for a Clair deployment in an OpenShift Container Platform cluster by entering the following command:

    $ oc -n quay-enterprise exec example-registry-clair-app-64dd48f866-6ptgw -- cat /usr/bin/clairctl > clairctl
  2. Set the permissions of the clairctl file so that it can be executed and run by the user, for example:

    $ chmod u+x ./clairctl
Retrieving and decoding the Clair configuration secret for Clair deployments on OpenShift Container Platform

To prepare Clair for a disconnected OpenShift Container Platform environment, you can retrieve and decode the Clair configuration secret, then set disable_updaters and airgap to true in clair-config.yaml.

Prerequisites
  • You have installed the clairctl command line utility tool.

Procedure
  1. Enter the following command to retrieve and decode the configuration secret, and then save it to a Clair configuration YAML:

    $ oc get secret -n quay-enterprise example-registry-clair-config-secret  -o "jsonpath={$.data['config\.yaml']}" | base64 -d > clair-config.yaml
  2. Update the clair-config.yaml file so that the disable_updaters and airgap parameters are set to True, for example:

    # ...
    indexer:
      airgap: true
    # ...
    matcher:
      disable_updaters: true
    # ...
Exporting the updaters bundle from a connected Clair instance

To export vulnerability database updates from a connected Clair instance, you can run clairctl export-updaters with your configuration file.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have retrieved and decoded the Clair configuration secret, and saved it to a Clair config.yaml file.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

Procedure
  • From a Clair instance that has access to the internet, use the clairctl CLI tool with your configuration file to export the updaters bundle. For example:

    $ ./clairctl --config ./config.yaml export-updaters updates.gz
Configuring access to the Clair database in the disconnected OpenShift Container Platform cluster

To access the Clair database in a disconnected OpenShift Container Platform cluster, you can identify the database service, forward its port, and point your Clair configuration at localhost.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have retrieved and decoded the Clair configuration secret, and saved it to a Clair config.yaml file.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

  • You have exported the updaters bundle from a Clair instance that has access to the internet.

Procedure
  1. Determine your Clair database service by using the oc CLI tool, for example:

    $ oc get svc -n quay-enterprise
    Example output
    NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                             AGE
    example-registry-clair-app            ClusterIP      172.30.224.93    <none>        80/TCP,8089/TCP                     4d21h
    example-registry-clair-postgres       ClusterIP      172.30.246.88    <none>        5432/TCP                            4d21h
    ...
  2. Forward the Clair database port so that the local machine can access it. For example:

    $ oc port-forward -n quay-enterprise service/example-registry-clair-postgres 5432:5432
  3. Update your Clair config.yaml file, for example:

    indexer:
        connstring: host=localhost port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
        layer_scan_concurrency: 5
        migrations: true
        scanlock_retry: 10
        airgap: true
        scanner:
          repo:
            rhel-repository-scanner:
              repo2cpe_mapping_file: /data/repository-to-cpe.json
          package:
            rhel_containerscanner:
              name2repos_mapping_file: /data/container-name-repos-map.json

    where:

    indexer.connstring

    Specifies the connection string for the database.

    indexer.scanner.repo.rhel-repository-scanner

    Specifies the repository scanner configuration.

    indexer.scanner.package.rhel_containerscanner

    Specifies the container scanner configuration.

Importing the updaters bundle into the disconnected OpenShift Container Platform cluster

To load vulnerability data into Clair on a disconnected OpenShift Container Platform cluster, you can run clairctl import-updaters with your configuration file and updaters bundle.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have retrieved and decoded the Clair configuration secret, and saved it to a Clair config.yaml file.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

  • You have exported the updaters bundle from a Clair instance that has access to the internet.

  • You have transferred the updaters bundle into your disconnected environment.

Procedure
  • Use the clairctl CLI tool to import the updaters bundle into the Clair database on OpenShift Container Platform. For example:

    $ ./clairctl --config ./clair-config.yaml import-updaters updates.gz
Set up a self-managed deployment of Clair for a disconnected OpenShift Container Platform cluster

To install the clairctl command-line utility for a self-managed Clair deployment on OpenShift Container Platform, you can copy the binary from a Clair container with Podman and make it executable.

Procedure
  1. Install the clairctl program for a self-managed Clair deployment by using the podman cp command, for example:

    $ sudo podman cp clairv4:/usr/bin/clairctl ./clairctl
  2. Set the permissions of the clairctl file so that it can be executed and run by the user, for example:

    $ chmod u+x ./clairctl
Deploying a self-managed Clair container for disconnected OpenShift Container Platform clusters

To deploy a self-managed Clair container for disconnected OpenShift Container Platform clusters, you can create a configuration directory, set disable_updaters and airgap in the Clair configuration file, and start the container with Podman.

Prerequisites
  • You have installed the clairctl command line utility tool.

Procedure
  1. Create a folder for your Clair configuration file, for example:

    $ mkdir /etc/clairv4/config/
  2. Create a Clair configuration file with the disable_updaters parameter set to True, for example:

    ---
    indexer:
      airgap: true
    ---
    matcher:
      disable_updaters: true
    ---
  3. Start Clair by using the container image, mounting in the configuration from the file you created:

    $ sudo podman run -it --rm --name clairv4 \
    -p 8081:8081 -p 8088:8088 \
    -e CLAIR_CONF=/clair/config.yaml \
    -e CLAIR_MODE=combo \
    -v /etc/clairv4/config:/clair:Z \
    {productrepo}/{clairimage}:{productminv}
Exporting the updaters bundle from a connected Clair instance

To export vulnerability database updates from a connected self-managed Clair instance, you can run clairctl export-updaters with your configuration file.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have deployed Clair.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

Procedure
  • From a Clair instance that has access to the internet, use the clairctl CLI tool with your configuration file to export the updaters bundle. For example:

    $ ./clairctl --config ./config.yaml export-updaters updates.gz
Configuring access to the Clair database in the disconnected OpenShift Container Platform cluster

To access the Clair database in a disconnected OpenShift Container Platform cluster for a self-managed deployment, you can identify the database service, forward its port, and point your Clair configuration at localhost.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have deployed Clair.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

  • You have exported the updaters bundle from a Clair instance that has access to the internet.

Procedure
  1. Determine your Clair database service by using the oc CLI tool, for example:

    $ oc get svc -n quay-enterprise
    Example output
    NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                             AGE
    example-registry-clair-app            ClusterIP      172.30.224.93    <none>        80/TCP,8089/TCP                     4d21h
    example-registry-clair-postgres       ClusterIP      172.30.246.88    <none>        5432/TCP                            4d21h
    ...
  2. Forward the Clair database port so that the local machine can access it. For example:

    $ oc port-forward -n quay-enterprise service/example-registry-clair-postgres 5432:5432
  3. Update your Clair config.yaml file, for example:

    indexer:
        connstring: host=localhost port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
        layer_scan_concurrency: 5
        migrations: true
        scanlock_retry: 10
        airgap: true
        scanner:
          repo:
            rhel-repository-scanner:
              repo2cpe_mapping_file: /data/repository-to-cpe.json
          package:
            rhel_containerscanner:
              name2repos_mapping_file: /data/container-name-repos-map.json

    where:

    indexer.connstring

    Specifies the connection string for the database.

    indexer.scanner.repo.rhel-repository-scanner
    indexer.scanner.package.rhel_containerscanner

    Specifies the container scanner configuration.

Importing the updaters bundle into the disconnected OpenShift Container Platform cluster

To load vulnerability data into a self-managed Clair deployment on a disconnected OpenShift Container Platform cluster, you can run clairctl import-updaters with your configuration file and updaters bundle.

Prerequisites
  • You have installed the clairctl command line utility tool.

  • You have deployed Clair.

  • The disable_updaters and airgap parameters are set to True in your Clair config.yaml file.

  • You have exported the updaters bundle from a Clair instance that has access to the internet.

  • You have transferred the updaters bundle into your disconnected environment.

Procedure
  • Use the clairctl CLI tool to import the updaters bundle into the Clair database on OpenShift Container Platform:

    $ ./clairctl --config ./clair-config.yaml import-updaters updates.gz

Common Product Enumeration mapping in Clair

Clair uses Common Product Enumeration (CPE) mapping files to map RPM packages to security data for Red Hat Enterprise Linux (RHEL) container images. Accurate vulnerability reports depend on these mapping files being available to the Clair scanner.

The scanner requires the CPE file to be present and accessible to process RPM packages properly. If these files are missing or inaccessible, RPM packages installed in the container image are skipped during the scanning process.

By default, the Clair indexer includes the repos2cpe and names2repos data files within the Clair container. This allows you to reference local paths such as /data/repository-to-cpe.json without additional external configuration.

Important

While Red Hat Product Security updates CPE files regularly, the versions bundled within the Clair container are only updated during Project Quay releases. This can lead to temporary discrepancies between the latest security data and the versions bundled with your current installation.

CPE mapping configuration reference

Common Product Enumeration (CPE) mapping configuration defines the fields and file paths used by Clair to associate packages with standardized product identifiers.

Table 1. Clair CPE mapping files
CPE Type Link to JSON mapping file

repos2cpe

Red Hat Repository-to-CPE JSON

names2repos

Red Hat Name-to-Repos JSON

Example configuration
indexer:
  scanner:
    repo:
      rhel-repository-scanner:
        repo2cpe_mapping_file: /data/repository-to-cpe.json
    package:
      rhel_containerscanner:
        name2repos_mapping_file: /data/container-name-repos-map.json

where:

repo2cpe_mapping_file

Specifies the path to the JSON file mapping Red Hat repositories to CPEs.

name2repos_mapping_file

Specifies the path to the JSON file mapping container names to repositories.

Resize Managed Storage

To expand storage capacity for your Red Hat Quay on OpenShift Container Platform deployment, you can use the OpenShift Container Platform console to resize the PostgreSQL and Clair PostgreSQL persistent volume claims (PVCs).

Resizing PostgreSQL and Clair PostgreSQL PVCs lets you increase storage beyond the default 50 GiB allocation when your registry needs more space.

When deploying Red Hat Quay on OpenShift Container Platform, three distinct PVCs are deployed:

  • One for the PostgreSQL 15 registry.

  • One for the Clair PostgreSQL 15 registry.

  • One that uses NooBaa as a backend storage.

Note

The connection between Project Quay and NooBaa is done through the S3 API and ObjectBucketClaim API in OpenShift Container Platform. Project Quay leverages that API group to create a bucket in NooBaa, obtain access keys, and automatically set everything up. On the backend, or NooBaa, side, that bucket is creating inside of the backing store. As a result, NooBaa PVCs are not mounted or connected to Project Quay pods.

Prerequisites
  • You have cluster admin privileges on OpenShift Container Platform.

Procedure
  1. Log into the OpenShift Container Platform console and select Storage → Persistent Volume Claims.

  2. Select the desired PersistentVolumeClaim for either PostgreSQL 13 or Clair PostgreSQL 13, for example, example-registry-quay-postgres-13.

  3. From the Action menu, select Expand PVC.

  4. Enter the new size of the Persistent Volume Claim and select Expand.

    After a few minutes, the expanded size should reflect in the PVC’s Capacity field.

Customizing Default Operator Images

Note

Currently, customizing default Operator images is not supported on IBM Power and IBM Z.

Customizing default Operator images lets you override the default container images used by the Project Quay Operator by setting environment variables in the ClusterServiceVersion object.

Important

Customizing default Operator images is not supported for production Project Quay environments and is only recommended for development or testing purposes. There is no guarantee your deployment will work correctly when using non-default images with the Project Quay Operator.

Environment Variables

The Project Quay Operator uses environment variables to override default container images for components such as base, clair, postgres, and redis. You can set these variables in the ClusterServiceVersion object to customize which images the Operator uses for each component.

Table 2. ClusterServiceVersion environment variables

Environment Variable

Component

RELATED_IMAGE_COMPONENT_QUAY

base

RELATED_IMAGE_COMPONENT_CLAIR

clair

RELATED_IMAGE_COMPONENT_POSTGRES

postgres and clair databases

RELATED_IMAGE_COMPONENT_REDIS

redis

Note

Overridden images must be referenced by manifest (@sha256:) and not by tag (:latest).

Apply overrides to a running Operator

To override container images for a running Project Quay Operator, you can modify the ClusterServiceVersion object to add environment variables that point to your custom images. This applies the overrides at the Operator level, so all QuayRegistry instances use the same custom images.

Procedure
  1. The ClusterServiceVersion object is Operator Lifecycle Manager’s representation of a running Operator in the cluster. Find the Project Quay Operator’s ClusterServiceVersion by using a Kubernetes UI or the kubectl/oc CLI tool. For example:

    $ oc get clusterserviceversions -n <namespace>
  2. Using the UI, oc edit, or another method, modify the ClusterServiceVersion object to include the environment variables outlined above to point to the override images:

    JSONPath: spec.install.spec.deployments[0].spec.template.spec.containers[0].env

    - name: RELATED_IMAGE_COMPONENT_QUAY
      value: quay.io/projectquay/quay@sha256:c35f5af964431673f4ff5c9e90bdf45f19e38b8742b5903d41c10cc7f6339a6d
    - name: RELATED_IMAGE_COMPONENT_CLAIR
      value: quay.io/projectquay/clair@sha256:70c99feceb4c0973540d22e740659cd8d616775d3ad1c1698ddf71d0221f3ce6
    - name: RELATED_IMAGE_COMPONENT_POSTGRES
      value: centos/postgresql-10-centos7@sha256:de1560cb35e5ec643e7b3a772ebaac8e3a7a2a8e8271d9e91ff023539b4dfb33
    - name: RELATED_IMAGE_COMPONENT_REDIS
      value: centos/redis-32-centos7@sha256:06dbb609484330ec6be6090109f1fa16e936afcf975d1cbc5fff3e6c7cae7542

AWS S3 CloudFront

To configure AWS S3 CloudFront for your Project Quay backend registry storage, you can create a secret that includes your config.yaml file and the CloudFront signing key. This enables CloudFront content delivery for your registry storage.

Procedure
  • Create a secret that includes your config.yaml file and the CloudFront signing key by entering the following command:

    $ oc create secret generic --from-file config.yaml=./config_awss3cloudfront.yaml --from-file default-cloudfront-signing-key.pem=./default-cloudfront-signing-key.pem test-config-bundle

Geo-replication

Geo-replication connects multiple geographically distributed Project Quay deployments so that clients use them as a single registry. Standalone and Operator-based deployments support geo-replication.

With geo-replication, regions share one database and one Redis instance while each region keeps local object storage. Clients push and pull through a common entrypoint, typically a global load balancer, and blob data replicates asynchronously between storage backends.

Geo-replication features

With geo-replication in Project Quay, image pushes go to the nearest storage backend and data replicates in the background so that pulls can use the closest available engine.

The following are the key features of geo-replication:

  • When geo-replication is configured, container image pushes are written to the preferred storage engine for that Project Quay instance. This is typically the nearest storage backend within the region.

  • After the initial push, image data is replicated in the background to other storage engines.

  • The list of replication locations is configurable and those can be different storage backends.

  • An image pull always uses the closest available storage engine to maximize pull performance.

  • If replication has not been completed yet, the pull uses the source storage backend instead.

Geo-replication requirements and constraints

Project Quay geo-replication requires shared storage, database, Redis, network access between regions, and a global load balancer. Review these constraints before you deploy.

The following are the requirements and constraints for geo-replication:

  • In geo-replicated setups, Project Quay requires that all regions are able to read and write to all other regions' object storage. Object storage must be geographically accessible by all other regions.

  • Project Quay has no internal awareness of object storage health. Plan a global load balancer that monitors site and storage health and redirects clients when a site must be taken offline.

  • Geo-replication does not automatically fail over object storage. If one site’s storage is unavailable, you must shut down that site’s Project Quay deployment so the load balancer can send clients to healthy sites. Otherwise, clients experience pull and push failures.

  • Geo-replication is asynchronous. Permanent loss of a site can lose blobs that were stored there but not yet replicated.

  • A single database, and therefore all metadata and Project Quay configuration, is shared across all regions. Geo-replication does not replicate the database and does not fail over to another database.

  • A single Redis cache is shared across the entire Project Quay setup and must be accessible by all Project Quay pods.

  • Use the same configuration across regions, except for the preferred storage backend per region.

  • Geo-replication requires object storage in each region. It does not work with local storage.

  • Each region must be able to access every storage engine in each region, which requires a network path. A storage proxy is an alternative when direct paths are not practical.

  • The entire storage backend, for example, all blobs, is replicated. Repository mirroring, by contrast, can be limited to a repository or an image.

  • All Project Quay instances must share the same entrypoint, typically through a load balancer.

  • All Project Quay instances must have the same set of superusers, as they are defined inside the common configuration file.

  • For Operator-based geo-replication, plan a shared Clair database (typically an unmanaged Clair database) so every region uses the same vulnerability data store.

  • Geo-replication requires SSL/TLS certificates and keys.

If the above requirements cannot be met, you should instead use two or more distinct Project Quay deployments and take advantage of repository mirroring functions.

Preparing your OpenShift Container Platform environment for geo-replication

To prepare your OpenShift Container Platform environment for Project Quay geo-replication, you can deploy shared PostgreSQL and Redis instances, create object storage backends, and configure a load balancer.

Procedure
  1. Deploy a PostgreSQL instance for Project Quay.

  2. Log in to the database by entering the following command:

    psql -U <username> -h <hostname> -p <port> -d <database_name>
  3. Create a database for Project Quay named quay. For example:

    CREATE DATABASE quay;
  4. Enable the pg_trgm extension inside the database:

    \c quay;
    CREATE EXTENSION IF NOT EXISTS pg_trgm;
  5. Deploy a Redis instance:

    Note
    • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.

    • Deploying a Redis instance is required if you are leveraging Builders.

    1. Deploy a VM for Redis.

    2. Verify that the Redis instance is accessible from the clusters where Project Quay is running.

    3. Port 6379/TCP must be open.

    4. Run Redis inside the instance:

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis
  6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket is close to the first, or primary, cluster, and the other runs closer to the second, or secondary, cluster.

  7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.

  8. Configure a load balancer to provide a single entry point to the clusters.

Configuring geo-replication for Project Quay on OpenShift Container Platform

To configure geo-replication for Project Quay on OpenShift Container Platform, you can create a shared config.yaml file with PostgreSQL, Redis, and storage backend details, create a configBundleSecret, and deploy QuayRegistry resources in each cluster with storage preference overrides. This enables multiple Project Quay deployments to work as a single registry with improved performance across geographic regions.

Prerequisites
  • You have prepared your OpenShift Container Platform environment for geo-replication by following the "Preparing your OpenShift Container Platform environment for geo-replication" procedure.

Procedure
  1. Create a config.yaml file that is shared between clusters. This config.yaml file contains the details for the common PostgreSQL, Redis and storage backends:

    Geo-replication config.yaml file
    SERVER_HOSTNAME: <georep.quayteam.org or any other name>
    DB_CONNECTION_ARGS:
      autorollback: true
      threadlocals: true
    DB_URI: postgresql://postgres:password@10.19.0.1:5432/quay
    BUILDLOGS_REDIS:
      host: 10.19.0.2
      port: 6379
    USER_EVENTS_REDIS:
      host: 10.19.0.2
      port: 6379
    DATABASE_SECRET_KEY: 0ce4f796-c295-415b-bf9d-b315114704b8
    DISTRIBUTED_STORAGE_CONFIG:
      usstorage:
        - GoogleCloudStorage
        - access_key: GOOGQGPGVMASAAMQABCDEFG
          bucket_name: georep-test-bucket-0
          secret_key: AYWfEaxX/u84XRA2vUX5C987654321
          storage_path: /quaygcp
      eustorage:
        - GoogleCloudStorage
        - access_key: GOOGQGPGVMASAAMQWERTYUIOP
          bucket_name: georep-test-bucket-1
          secret_key: AYWfEaxX/u84XRA2vUX5Cuj12345678
          storage_path: /quaygcp
    DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - usstorage
      - eustorage
    DISTRIBUTED_STORAGE_PREFERENCE:
      - usstorage
      - eustorage
    FEATURE_STORAGE_REPLICATION: true

    where:

    SERVER_HOSTNAME

    Specifies the hostname of the global load balancer. Must match the hostname of the global load balancer.

  2. Create the configBundleSecret custom resource (CR) by entering the following command:

    $ oc create secret generic --from-file config.yaml=./config.yaml georep-config-bundle
  3. In each of the clusters, set the configBundleSecret and use the QUAY_DISTRIBUTED_STORAGE_PREFERENCE environmental variable override to configure the appropriate storage for that cluster. For example:

    Note

    The config.yaml file between both deployments must match. If making a change to one cluster, it must also be changed in the other.

    US cluster QuayRegistry example
    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: georep-config-bundle
      components:
        - kind: objectstorage
          managed: false
        - kind: route
          managed: true
        - kind: tls
          managed: false
        - kind: postgres
          managed: false
        - kind: clairpostgres
          managed: false
        - kind: redis
          managed: false
        - kind: quay
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: usstorage
        - kind: mirror
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: usstorage
    Note

    Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL/TLS and Routes.

    European cluster
    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: georep-config-bundle
      components:
        - kind: objectstorage
          managed: false
        - kind: route
          managed: true
        - kind: tls
          managed: false
        - kind: postgres
          managed: false
        - kind: clairpostgres
          managed: false
        - kind: redis
          managed: false
        - kind: quay
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: eustorage
        - kind: mirror
          managed: true
          overrides:
            env:
            - name: QUAY_DISTRIBUTED_STORAGE_PREFERENCE
              value: eustorage
    Note

    Because SSL/TLS is unmanaged, and the route is managed, you must supply the certificates directly in the config bundle. For more information, see Configuring SSL and TLS for Project Quay.

Mixed storage for geo-replication

Geo-replication in Project Quay can use mixed storage backends, such as public-cloud object storage and on-premise Ceph, when you secure cross-site access appropriately.

Because geo-replication supports multiple replication targets, Red Hat recommends that you use a VPN or token pair with bucket-specific access to meet security requirements. This results in the public cloud instance of Project Quay having access to on-premise storage, but the network is encrypted, protected, and uses ACLs, thereby meeting security requirements. If you cannot implement these security measures, it might be preferable to deploy two distinct Project Quay registries and to use repository mirroring as an alternative to geo-replication.

Upgrade Operator geo-replication

To upgrade your geo-replicated Red Hat Quay on OpenShift Container Platform deployment, you must stop operations, scale down secondary systems, upgrade the primary system, then upgrade secondary systems. This ensures a safe upgrade process with minimal downtime across your geo-replicated registry.

Important
  • When upgrading geo-replicated Red Hat Quay on OpenShift Container Platform deployment to the next y-stream release (for example, Project Quay {producty-n1} → Project Quay {producty}), you must stop operations before upgrading.

  • There is intermittent downtime down upgrading from one y-stream release to the next.

  • It is highly recommended to back up your Red Hat Quay on OpenShift Container Platform deployment before upgrading.

The following procedure assumes that you are running the Project Quay registry on three or more systems. For this procedure, three systems named System A, System B, and System C are used. System A serves as the primary system in which the Project Quay Operator is deployed.

Procedure
  1. On System B and System C, scale down your Project Quay registry. This is done by disabling auto scaling and overriding the replica county for Project Quay, mirror workers, and Clair if it is managed. Note that you must keep the Project Quay registry running on System A. Do not update the quayregistry.yaml file on System A.

    Use the following quayregistry.yaml file as a reference:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: false
        - kind: quay
          managed: true
          overrides:
            replicas: 0
        - kind: clair
          managed: true
          overrides:
            replicas: 0
        - kind: mirror
          managed: true
          overrides:
            replicas: 0
        …

    where:

    managed: false

    Disables auto scaling of Quay, Clair and Mirroring workers

    overrides

    Sets the replica count to 0 for components accessing the database and objectstorage

  2. Wait for the registry-quay-app, registry-quay-mirror, and registry-clair-app pods to disappear. Enter the following command to check their status:

    oc get pods -n <quay-namespace>
    Example output
    quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
    quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
    quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
    quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m
  3. On System A, initiate a Project Quay upgrade to the latest y-stream version. This is a manual process. For more information about upgrading installed Operators, see Upgrading installed Operators. For more information about Project Quay upgrade paths, see Upgrading the Project Quay Operator.

  4. After the new Project Quay registry is installed, the necessary upgrades on the cluster are automatically completed. Afterwards, new Project Quay pods are started with the latest y-stream version. Additionally, new Quay pods are scheduled and started.

  5. Confirm that the update has properly worked by navigating to the Project Quay UI:

    1. In the OpenShift console, navigate to Operators → Installed Operators, and click the Registry Endpoint link.

      Important

      Do not execute the following step until the Project Quay UI is available. Do not upgrade the Project Quay registry on System B and on System C until the UI is available on System A.

  6. Confirm that the update has properly worked on System A, initiate the Project Quay upgrade on System B and on System C. The Operator upgrade results in an upgraded Project Quay installation, and the pods are restarted.

    Note

    Because the database schema is correct for the new y-stream installation, the new pods on System B and on System C should quickly start.

  7. After updating, revert the changes made in step 1 of this procedure by removing overrides for the components. For example:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: true
        - kind: quay
          managed: true
        - kind: clair
          managed: true
        - kind: mirror
          managed: true
        …

    where:

    kind: horizontalpodautoscaler:: Set this resource to True if the horizontalpodautoscaler resource was set to True before the upgrade procedure, or if you want Project Quay to scale in case of a resource shortage.

Remove a geo-replicated site from your Red Hat Quay on OpenShift Container Platform deployment

To remove a geo-replicated site from a Red Hat Quay on OpenShift Container Platform deployment, you can sync blobs between sites, update storage configuration, and run the removelocation utility.

Prerequisites
  • You are logged into OpenShift Container Platform.

  • You have configured Project Quay geo-replication with at least two sites, for example, usstorage and eustorage.

  • Each site has its own Organization, Repository, and image tags.

Procedure
  1. Sync the blobs between all of your defined sites by running the following command:

    $ python -m util.backfillreplication
    Warning

    Prior to removing storage engines from your Project Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

    When running this command, replication jobs are created which are picked up by the replication worker. If blobs need to be replicated, the script returns UUIDs of blobs that are replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that no more blobs remain to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

    Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

    This step must be completed before proceeding.

  2. In your Project Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

  3. Identify your Project Quay application pods by entering the following command:

    $ oc get pod -n <quay_namespace>
    Example output:
    quay390usstorage-quay-app-5779ddc886-2drh2
    quay390eustorage-quay-app-66969cd859-n2ssm
  4. Open an interactive shell session in the usstorage pod by entering the following command:

    $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2
  5. Permanently remove the eustorage site by entering the following command:

    Important

    The following action cannot be undone. Use with caution.

    sh-4.4$ python -m util.removelocation eustorage
    Example output:
    WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
    Deleted placement 30
    Deleted placement 31
    Deleted placement 32
    Deleted placement 33
    Deleted location eustorage

Backing up and restoring Project Quay managed by the Project Quay Operator

To protect Project Quay data and enable recovery from failures, you can back up and restore Project Quay instances managed by the Project Quay Operator on OpenShift Container Platform.

Enabling read-only mode for Red Hat Quay on OpenShift Container Platform

Read-only mode keeps your Red Hat Quay on OpenShift Container Platform registry online during backup and restore by blocking write operations. You can use this mode when scaling down the deployment is unacceptable.

When backing up and restoring, you are required to scale down your Red Hat Quay on OpenShift Container Platform deployment. This results in service unavailability during the backup period which, in some cases, might be unacceptable. Enabling read-only mode ensures service availability during the backup and restore procedure for Red Hat Quay on OpenShift Container Platform deployments.

Note

In some cases, you cannot enable read-only mode for Project Quay because it requires inserting a service key and other manual configuration changes. As an alternative to read-only mode, Project Quay administrators might consider enabling the DISABLE_PUSHES feature. When this field is set to True, users are unable to push images or image tags to the registry when using the CLI. Enabling DISABLE_PUSHES differs from read-only mode because the database is not set as read-only when it is enabled.

This field might be useful in some situations such as when Project Quay administrators want to calculate their registry’s quota and disable image pushing until after calculation has completed. With this method, administrators can avoid putting the whole registry in read-only mode, which affects the database, so that most operations can still be done.

There are two methods to enable read-only mode for Red Hat Quay on OpenShift Container Platform:

Operator-managed method

The Operator-managed method is available in Project Quay {producty} and later. It allows the Project Quay Operator to manage the read-only lifecycle for Red Hat Quay on OpenShift Container Platform deployments. This method is the preferred method for enabling read-only mode for Red Hat Quay on OpenShift Container Platform.

Manual method

The manual method is the traditional method of enabling read-only mode for Red Hat Quay on OpenShift Container Platform. It requires you to manually insert a service key and make configuration changes. When the Operator detects REGISTRY_STATE: readonly set manually in the configuration, it defers image version upgrades and database migration jobs and sets a ManualReadOnlyDetected status condition.

  • If you are using Red Hat Enterprise Linux (RHEL) 8:

    • You have installed Python 3 on your machine.

    • You have downloaded the python3-virtualenv package.

    • You have installed the git CLI.

  • You have cloned the quay/quay repository.

  • You have installed the oc CLI.

  • You have access to the cluster with cluster-admin privileges.

Important

The two methods are mutually exclusive. If REGISTRY_STATE: readonly or any INSTANCE_SERVICE_KEY_* key is present in your config Secret, the Operator blocks operator-managed read-only mode with a ManualMigrationRequired condition. If the conflict is in a QUAY_OVERRIDE_CONFIG environment variable override, the condition reason is OverrideConflict. Remove the conflicting keys before using the Operator-managed method.

Additional resources

Enable Operator-managed read-only mode for Red Hat Quay on OpenShift Container Platform

Use the Project Quay Operator to manage the full read-only lifecycle for application-consistent backups.

By setting spec.readOnly: true on the QuayRegistry custom resource, the Operator automatically handles service key creation, configuration rendering, deployment restarts, and HPA management without requiring any manual configuration changes.

This procedure replaces the manual steps of creating service keys and editing the configuration secret that were required in earlier releases.

Note

The Operator-managed method and manual method are mutually exclusive. If REGISTRY_STATE: readonly or any INSTANCE_SERVICE_KEY_* key is present in your config Secret or config bundle, the Operator sets a ManualMigrationRequired condition and does not enter the operator-managed read-only state machine. If the conflict is in a QUAY_OVERRIDE_CONFIG environment variable override, the condition reason is OverrideConflict. Remove any conflicting keys before setting spec.readOnly: true.

Prerequisites
  • You have a QuayRegistry resource deployed and in a healthy state.

  • You have cluster-admin or equivalent permissions on the OpenShift Container Platform cluster.

  • The QuayRegistry is running Project Quay {producty} or later (this feature requires the matching Quay application image).

  • You do not have REGISTRY_STATE set as a manual override in your QuayRegistry configuration.

Procedure
  1. Enter read-only mode by patching the QuayRegistry custom resource:

    $ oc patch quayregistry <registry_name> -n <namespace> \
      --type=merge \
      -p '{"spec":{"readOnly":true}}'
  2. Monitor the read-only lifecycle status until the phase reaches ReadOnly:

    $ oc get quayregistry <registry_name> -n <namespace> \
      -o jsonpath='{.status.conditions[?(@.type=="ReadOnly")]}'

    The Operator transitions through the following phases in order:

    • PreparingKey: During this phase, the Operator creates the service key Secret and waits for Project Quay to import it.

    • EnteringReadOnly: During this phase, the Operator renders REGISTRY_STATE: readonly in the configuration and restarts the deployment.

    • ReadOnly: During this phase, the registry is fully in read-only mode. Writes are blocked.

      Note

      If the QuayRegistry status shows a ManualMigrationRequired condition, REGISTRY_STATE: readonly or an INSTANCE_SERVICE_KEY_* key is present in your config Secret or config bundle. If the condition reason is OverrideConflict, the conflicting key is in a QUAY_OVERRIDE_CONFIG environment variable override. In either case, remove the conflicting key and re-apply the patch.

  3. Perform your backup operation while the registry is in read-only mode. For more information, see "Backing up and restoring Red Hat Quay on OpenShift Container Platform".

  4. When your backup is complete, disable read-only mode. For more information, see Enable read-only mode for Red Hat Quay on OpenShift Container Platform.

Disable Operator-managed read-only mode for Red Hat Quay on OpenShift Container Platform

After your backup operation is complete, disable Operator-managed read-only mode by setting spec.readOnly: false on the QuayRegistry custom resource.

Prerequisites
  • You have completed your backup operation while the registry was in read-only mode.

  • You have cluster-admin or equivalent permissions on the OpenShift Container Platform cluster.

Procedure
  1. Exit read-only mode when the backup is complete:

    $ oc patch quayregistry <registry_name> -n <namespace> \
      --type=merge \
      -p '{"spec":{"readOnly":false}}'
  2. Monitor the status until the registry returns to normal operation:

    $ oc get quayregistry <registry_name> -n <namespace> \
      -o jsonpath='{.status.conditions[?(@.type=="Available")]}'

    The Operator transitions through the ExitingReadOnly phase, cleans up the service key Secret, and restores normal deployment behavior.

Verification
  • Confirm that the ReadOnly condition is no longer present on the QuayRegistry resource:

    $ oc get quayregistry <registry_name> -n <namespace> -o yaml | grep -A5 "conditions:"
  • Confirm that image pushes are accepted:

    $ podman push <registry_route>/<namespace>/<repository>:<tag>

Manually enabling read-only mode for Red Hat Quay on OpenShift Container Platform

For deployments that do not use the Operator-managed read-only lifecycle, you can enable read-only mode manually by setting REGISTRY_STATE: readonly in the configuration secret. The Project Quay Operator detects this setting and defers upgrades automatically.

Note

The manual method requires direct modification of secrets inside your OpenShift Container Platform cluster and several manual steps. If you are running Project Quay {producty} or later, the preferred method is the Operator-managed approach using the spec.readOnly parameter.

The Operator-managed method and manual method are mutually exclusive. If REGISTRY_STATE: readonly or any INSTANCE_SERVICE_KEY_* key is present in your config Secret, setting spec.readOnly: true is blocked with a ManualMigrationRequired condition. If the conflict is in a QUAY_OVERRIDE_CONFIG environment variable override, the condition reason is OverrideConflict. Remove the conflicting keys before using the Operator-managed method.

Prerequisites
  • If you are using Red Hat Enterprise Linux (RHEL) 7.x:

    • You have enabled the Red Hat Software Collections List (RHSCL).

    • You have installed Python 3.6.

    • You have downloaded the virtualenv package.

    • You have installed the git CLI.

  • If you are using Red Hat Enterprise Linux (RHEL) 8:

    • You have installed Python 3 on your machine.

    • You have downloaded the python3-virtualenv package.

    • You have installed the git CLI.

  • You have cloned the quay/quay repository.

  • You have installed the oc CLI.

  • You have access to the cluster with cluster-admin privileges.

Upgrade deferral behavior

When the Project Quay Operator detects REGISTRY_STATE: readonly set directly in the user-provided config Secret, the merged config bundle Secret, or a QUAY_OVERRIDE_CONFIG environment variable override, it performs the following actions:

  • Sets a ReadOnly status condition on the QuayRegistry resource with reason: ManualReadOnlyDetected.

  • Defers image version upgrades. The Operator freezes the current Quay image and does not roll out a new version.

  • Defers PostgreSQL and Clair PostgreSQL upgrade jobs and version migration triggers.

  • Continues reconciling all other components normally.

The following condition is set on the QuayRegistry resource:

- type: ReadOnly
  status: "True"
  reason: ManualReadOnlyDetected
  message: "Registry is in read-only mode via user config (REGISTRY_STATE).
            Operator upgrades are deferred. Consider using spec.readOnly
            for full operator-managed lifecycle."

The condition is automatically cleared when REGISTRY_STATE is removed from all user-controlled configuration sources. After it is cleared, the Operator resumes normal upgrade and reconciliation behavior.

Create service keys for Red Hat Quay on OpenShift Container Platform

To enable Project Quay to communicate with components and sign completed requests such as image scanning and login, you can create service keys. Access the Quay container pod and run the keypair generation script to create the necessary keys.

Procedure
  1. Enter the following command to obtain a list of Project Quay pods:

    $ oc get pods -n <namespace>
    Example output
    example-registry-clair-app-7dc7ff5844-4skw5           0/1     Error                    0             70d
    example-registry-clair-app-7dc7ff5844-nvn4f           1/1     Running                  0             31d
    example-registry-clair-app-7dc7ff5844-x4smw           0/1     ContainerStatusUnknown   6 (70d ago)   70d
    example-registry-clair-app-7dc7ff5844-xjnvt           1/1     Running                  0             60d
    example-registry-clair-postgres-547d75759-75c49       1/1     Running                  0             70d
    example-registry-quay-app-76c8f55467-52wjz            1/1     Running                  0             70d
    example-registry-quay-app-76c8f55467-hwz4c            1/1     Running                  0             70d
    example-registry-quay-app-upgrade-57ghs               0/1     Completed                1             70d
    example-registry-quay-database-7c55899f89-hmnm6       1/1     Running                  0             70d
    example-registry-quay-mirror-6cccbd76d-btsnb          1/1     Running                  0             70d
    example-registry-quay-mirror-6cccbd76d-x8g42          1/1     Running                  0             70d
    example-registry-quay-redis-85cbdf96bf-4vk5m          1/1     Running                  0             70d
  2. Open a remote shell session to the Quay container by entering the following command:

    $ oc rsh example-registry-quay-app-76c8f55467-52wjz
  3. Create the necessary service keys by entering the following command:

    sh-4.4$ python3 tools/generatekeypair.py quay-readonly
    Example output
    Writing public key to quay-readonly.jwk
    Writing key ID to quay-readonly.kid
    Writing private key to quay-readonly.pem
Add keys to the PostgreSQL database

To register read-only service keys with Project Quay, you can insert the key and approval records into the PostgreSQL database.

Prerequisites
  • You have created the service keys.

Procedure
  1. Enter the following command to enter your Project Quay database environment:

    $ podman exec -it postgresql-quay psql -U postgres -d quay
  2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

    quay=# select * from servicekeyapproval;
    Example output:
     id | approver_id |          approval_type           |       approved_date        | notes
    ----+-------------+----------------------------------+----------------------------+-------
      1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
      2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
      3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
      4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
      5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
    ...
  3. Add the service key to your Project Quay database by entering the following query:

    quay=# INSERT INTO servicekey
      (name, service, metadata, kid, jwk, created_date, expiration_date)
      VALUES ('quay-readonly',
               'quay',
               '{}',
               '<contents_of_.kid_file>',
               '<contents_of_.jwk_file>',
               '<created_date_of_read-only>',
               '<expiration_date_of_read-only>');
    Example output:
    INSERT 0 1
  4. Next, add the key approval with the following query:

    quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
      VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
               <include_notes_here_on_why_this_is_being_added>);
    Example output:
    INSERT 0 1
  5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

    UPDATE servicekey
    SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
    WHERE name = 'quay-readonly';
    UPDATE 1
Configure read-only mode Red Hat Quay on OpenShift Container Platform

To enable read-only mode in Project Quay and safely manage registry operations such as backup and restore, you can modify the configuration secret and restart the Quay container.

Important

Deploying Red Hat Quay on OpenShift Container Platform in read-only mode requires you to modify the secrets stored inside of your OpenShift Container Platform cluster. It is highly recommended that you create a backup of the secret prior to making changes to it.

Prerequisites
  • You have created the service keys and added them to your PostgreSQL database.

Procedure
  1. Read the secret name of your Red Hat Quay on OpenShift Container Platform deployment by entering the following command:

    $ oc get deployment -o yaml <quay_main_app_deployment_name>
  2. Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files by entering the following commands:

    $ base64 -w0 quay-readonly.kid
    Example output
    ZjUyNDFm...
    $ base64 -w0 quay-readonly.pem
    Example output
    <example_secret>...
  3. Obtain the current configuration bundle and secret by entering the following command. Save the output to a file called config.yaml:

    $ oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml
  4. Edit the config.yaml file and add the following information to enable read-only mode:

    # ...
    REGISTRY_STATE: readonly
    INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
    INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
    # ...
  5. Save the file and base64 encode it by entering the following command:

    $ base64 -w0 quay-config.yaml
  6. Scale down the Project Quay Operator pods to 0 by entering the following command. This ensures that the Operator does not reconcile the secret after editing it.

    $ oc scale --replicas=0 deployment quay-operator -n openshift-operators
  7. Edit the secret to include the new content by entering the following command:

    $ oc edit secret quay-config-secret-name -n quay-namespace
    # ...
    data:
      "quay-readonly.kid": "ZjUyNDFm..."
      "quay-readonly.pem": "<example_secret>..."
      "config.yaml": "QUNUSU9OX0xPR19..."
    # ...

    With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

Scale up the Project Quay from a read-only deployment

To exit read-only mode and restore normal operations in Project Quay, you can remove the read-only settings from the config.yaml file and scale the Operator deployment back up.

Note

Depending on your needs, you might wait to scale up the Project Quay deployment after backing up and restoring your registry.

Procedure
  1. Edit the config.yaml file and remove the following information:

    # ...
    REGISTRY_STATE: readonly
    INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
    INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
    # ...
  2. Scale the Project Quay Operator back up by entering the following command:

    $ oc scale --replicas=1 deployment quay-operator -n openshift-operators

Backing up Red Hat Quay on OpenShift Container Platform

To create backups of your Red Hat Quay on OpenShift Container Platform deployment for disaster recovery, you can back up the configuration, PostgreSQL database, and object storage. Regular backups ensure you can restore your registry to a previous state if needed.

Database backups should be performed regularly using either the supplied tools on the PostgreSQL image or your own backup infrastructure. The Project Quay Operator does not ensure that the PostgreSQL database is backed up.

Important

PostgreSQL and S3 object storage backups must be taken at the same time to avoid desynchronization. If backups are taken at different times, the database might contain references to storage blobs that are not present in the storage backup, which can cause data inconsistency and restore failures.

Note

This procedure covers backing up your Project Quay PostgreSQL database. It does not cover backing up the Clair PostgreSQL database. Backing up the Clair PostgreSQL database is not needed because it can be recreated. If you opt to recreate it from scratch, you wait for the information to be repopulated after all images inside of your Project Quay deployment are scanned. During this downtime, security reports are unavailable.

If you are considering backing up the Clair PostgreSQL database, you must consider that its size is dependent upon the number of images stored inside of Project Quay. As a result, the database can be extremely large.

Prerequisites for backing up Red Hat Quay on OpenShift Container Platform

  • A healthy Project Quay deployment on OpenShift Container Platform using the Project Quay Operator. The status condition Available is set to True.

  • The components quay, postgres and objectstorage are set to managed: true

  • If the component clair is set to managed: true the component clairpostgres is also set to managed: true (starting with Project Quay v3.7 or later)

Note

If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Project Quay deployment, you must refer to the service provider or vendor documentation to create a backup of the data. You can refer to the tools described in this guide as a starting point on how to backup your external PostgreSQL database or object storage.

Project Quay configuration backup

To back up your Project Quay configuration for disaster recovery, you can export the QuayRegistry custom resource, back up the managed secret keys, and save the config bundle and config.yaml files. This procedure creates backup files that you can use to restore your registry configuration.

Procedure
  1. To back the QuayRegistry custom resource by exporting it, enter the following command:

    $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml
  2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

      metadata.creationTimestamp
      metadata.finalizers
      metadata.generation
      metadata.resourceVersion
      metadata.uid
  3. Backup the managed keys secret by entering the following command:

    Note

    If you are running a version older than Project Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Project Quay for the first time. These are stored in a secret called <quay_registry_name>-quay-registry-managed-secret-keys in the namespace of the QuayRegistry resource.

    $ oc get secret -n <quay_namespace> <quay_registry_name>-quay-registry-managed-secret-keys -o yaml > managed_secret_keys.yaml
  4. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: <quay_registry_name>-quay-registry-managed-secret-keys
      namespace: <quay_namespace>
    data:
      CONFIG_EDITOR_PW: <redacted>
      DATABASE_SECRET_KEY: <redacted>
      DB_ROOT_PW: <redacted>
      DB_URI: <redacted>
      SECRET_KEY: <redacted>
      SECURITY_SCANNER_V4_PSK: <redacted>

    All information under the data property should remain the same.

  5. Redirect the current Quay configuration file by entering the following command:

    $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml
  6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

    $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml
  7. Obtain the Quay database name:

    $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
    Example output
    quayregistry-quay-database

Scale down the Project Quay deployment

To create a consistent backup of your Project Quay deployment, you must scale down the deployment by disabling auto scaling and setting replica counts to zero. This ensures the registry is in a quiescent state before backing up.

Important

This step is needed to create a consistent backup of the state of your Project Quay deployment. Do not omit this step, including in setups where PostgreSQL databases and/or S3-compatible object storage are provided by external services (unmanaged by the Project Quay Operator).

Procedure
  1. Scale down the Project Quay deployment by disabling auto scaling and overriding the replica count for Project Quay, mirror workers, and Clair (if managed). For example:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: false
        - kind: quay
          managed: true
          overrides:
            replicas: 0
        - kind: clair
          managed: true
          overrides:
            replicas: 0
        - kind: mirror
          managed: true
          overrides:
            replicas: 0
        …

    where:

    managed: false

    Disables auto scaling of Quay, Clair and Mirroring workers.

    overrides

    Sets the replica count to 0 for components accessing the database and objectstorage.

  2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Project Quay Operator) to disappear. You can check their status by entering the following command:

    $ oc get pods -n <quay_namespace>
    Example output:
    $ oc get pod
    Example output
    quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
    quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
    quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
    quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
    quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

Back up the Project Quay managed database

To back up your Project Quay managed database for disaster recovery, you can identify the PostgreSQL pod and use pg_dump to create a backup SQL file. This procedure creates a backup that you can use to restore your database.

Note

If your Project Quay deployment is configured with external, or unmanged, PostgreSQL database(s), refer to your vendor’s documentation on how to create a consistent backup of these databases.

Procedure
  1. Identify the Project Quay PostgreSQL pod name by entering the following command:

    $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'
    Example output:
    quayregistry-quay-database-59f54bb7-58xs7
  2. Download a backup database by entering the following command:

    $ oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

Back up the Project Quay managed object storage

To back up your Project Quay managed object storage for disaster recovery, you can export AWS credentials from secrets and use the aws s3 sync command to copy all blobs to a local directory. This procedure creates a backup of your registry’s object storage data.

The instructions in this section apply to the following configurations:

  • Standalone, multi-cloud object gateway configurations

  • OpenShift Data Foundations storage requires that the Project Quay Operator provisioned an S3 object storage bucket from, through the ObjectStorageBucketClaim API.

Note

You can also use rclone or sc3md instead of the AWS command line utility.

Procedure
  1. Decode and export the AWS_ACCESS_KEY_ID by entering the following command:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
  2. Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
  3. Create a new directory by entering the following command:

    $ mkdir blobs
  4. Copy all blobs to the directory by entering the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

Scale up the Project Quay deployment

To restore your Project Quay deployment to normal operation after scaling down, you can re-enable auto scaling and remove replica overrides for quay, mirror workers, and Clair. This restores your registry to full capacity after completing backup or maintenance tasks.

Procedure
  1. Scale up the Project Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. For example:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: true
        - kind: quay
          managed: true
        - kind: clair
          managed: true
        - kind: mirror
          managed: true
        …

    where:

    spec.components.horizontalpodautoscaler.managed

    Specifies re-enabling auto scaling of Quay, Clair and Mirroring workers again.

    spec.components.quay

    Specifies that replica overrides are removed again to scale the Quay components back up.

  2. Check the status of the Project Quay deployment by entering the following command:

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>
    Example output:
    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      ...
      name: registry
      namespace: <quay-namespace>
      ...
    spec:
      ...
    status:
      - lastTransitionTime: '2022-06-20T05:31:17Z'
        lastUpdateTime: '2022-06-20T17:31:13Z'
        message: All components reporting as healthy
        reason: HealthChecksPassing
        status: 'True'
        type: Available

Restoring Project Quay

To restore your Project Quay registry when the Operator manages the database, you can restore the configuration, database, and object storage from backups. This procedure restores your registry to a previous state after performing the backup process.

Prerequisites for restoring Project Quay

The following prerequisites are required to restore Project Quay:

  • Project Quay is deployed on OpenShift Container Platform using the Project Quay Operator.

  • A backup of the Project Quay configuration managed by the Project Quay Operator has been created following the instructions in the Backing up Project Quay section.

  • The object storage bucket used by Project Quay has been backed up.

  • The components quay, postgres and objectstorage are set to managed: true

  • If the component clair is set to managed: true, the component clairpostgres is also set to managed: true.

  • There is no running Project Quay deployment managed by the Project Quay Operator in the target namespace on your OpenShift Container Platform cluster

Note

If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Project Quay deployment, you must refer to the service provider or vendor documentation to restore their data from a backup prior to restore Project Quay

Restore Project Quay from a backup

To restore your Project Quay registry and configuration from a backup, you can restore the configuration bundle, managed secret keys, and QuayRegistry custom resource. This procedure restores your registry to a previous state using backup files created with the backup process.

Prerequisites
  • You have backed up your Project Quay registry and configuration.

  • You have the backup files config-bundle.yaml, managed-secret-keys.yaml, and quay-registry.yaml.

Procedure
  1. Restore the backed up Project Quay configuration by entering the following command:

    $ oc create -f ./config-bundle.yaml
    Important

    If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

  2. Restore the generated keys from the backup by entering the following command:

    $ oc create -f ./managed-secret-keys.yaml
  3. Restore the QuayRegistry custom resource by entering the following command:

    $ oc create -f ./quay-registry.yaml
  4. Check the status of the Project Quay deployment by entering the following command. Wait for it to be available:

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

Scale down the Project Quay deployment before restore

To scale down your Project Quay deployment, you can disable auto scaling and set replica counts to zero. This reduces resource consumption and stops registry operations temporarily.

Procedure
  1. Scale down the Project Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). For example:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: false
        - kind: quay
          managed: true
          overrides:
            replicas: 0
        - kind: clair
          managed: true
          overrides:
            replicas: 0
        - kind: mirror
          managed: true
          overrides:
            replicas: 0
        …

    where:

    spec.components.horizontalpodautoscaler.managed

    Specifies that the component is not managed by the Project Quay Operator.

    spec.components.quay.overrides

    Specifies the replica count for the component.

  2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Project Quay Operator) to disappear. You can check their status by running the following command:

    $ oc get pods -n <quay-namespace>

    Example output:

    registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
    registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
    registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

Restore your Project Quay database

To restore your Project Quay database from a backup, you can identify the database pod, upload the backup file, drop the existing database, and restore from the backup using psql. This procedure restores your database to a previous state using a backup SQL file.

Procedure
  1. Identify your Quay database pod by entering the following command:

    $ oc get pod -l quay-component=postgres -n  <quay_namespace> -o jsonpath='{.items[0].metadata.name}'
    Example output:
    quayregistry-quay-database-59f54bb7-58xs7
  2. Upload the backup by copying it from the local environment and into the pod by entering the following command:

    $ oc cp ./backup.sql -n <quay_namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
  3. Open a remote terminal to the database by entering the following command:

    $ oc rsh -n <quay_namespace> registry-quay-database-66969cd859-n2ssm
  4. Enter psql by entering the following command:

    bash-4.4$ psql
  5. You can list the database by entering the following command:

    postgres=# \l
    Example output
                                                      List of databases
               Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
    ----------------------------+----------------------------+----------+------------+------------+-----------------------
    postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
    quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |
  6. Drop the existing database by entering the following command:

    postgres=# DROP DATABASE "quayregistry-quay-database";
    Example output
    DROP DATABASE
  7. Exit the postgres CLI by entering the following command:

    \q
  8. Redirect your PostgreSQL database to your backup database by entering the following command:

    sh-4.4$ psql < /tmp/backup.sql
  9. Exit bash by entering the following command:

    sh-4.4$ exit

Restore the Project Quay object storage data

To restore your Project Quay object storage data from a backup, you can export AWS credentials from secrets and use the aws s3 sync command to upload blobs to your storage bucket. This procedure restores your registry’s object storage data using backup files.

Note

You can also use rclone or sc3md instead of the AWS command line utility.

Procedure
  1. Export the AWS_ACCESS_KEY_ID by entering the following command:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
  2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
  3. Upload all blobs to the bucket by running the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

Scale up the Project Quay deployment after restore

To scale up your Project Quay deployment, you can re-enable auto scaling and remove replica overrides. This restores your registry to normal operation after scaling down.

Procedure
  • Scale up the Project Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. For example:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: registry
      namespace: ns
    spec:
      components:
        …
        - kind: horizontalpodautoscaler
          managed: true
        - kind: quay
          managed: true
        - kind: clair
          managed: true
        - kind: mirror
          managed: true
        …

    where:

    spec.components.horizontalpodautoscaler.managed

    Specifies re-enabling auto scaling of Project Quay, Clair and mirroring workers again.

    spec.components.quay

    Specifies that replica overrides are removed again to scale the Project Quay components back up.

Volume size overrides

Volume size overrides let you specify the desired capacity and storage class for managed components that use persistent or ephemeral volumes in your Project Quay deployment. You can set larger volumes upfront for performance reasons or when your storage backend does not support resizing.

The default size for the managed postgres and clairpostgres persistent volumes is 50Gi. The default size for the managed clair ephemeral scratch volume is 20Gi.

Use overrides.volumeSize and overrides.storageClassName on the postgres, clairpostgres, and clair components in the QuayRegistry custom resource. For postgres and clairpostgres, set these overrides during initial installation of the component. Changing storageClassName after a database PVC is created is not supported.

Note

To expand database storage on an existing deployment, see Resizing managed storage.

In the following example, storage overrides are applied to three managed components:

  • On clair, volumeSize and storageClassName configure ephemeral scratch storage for image layer extraction at /var/tmp.

  • On postgres and clairpostgres, volumeSize and storageClassName configure the managed PostgreSQL database persistent volumes.

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: quay-example
  namespace: quay-enterprise
spec:
  configBundleSecret: config-bundle-secret
  components:
    - kind: objectstorage
      managed: false
    - kind: route
      managed: true
    - kind: tls
      managed: false
    - kind: clair
      managed: true
      overrides:
        volumeSize: 50Gi
        storageClassName: fast-ssd
    - kind: postgres
      managed: true
      overrides:
        volumeSize: 70Gi
        storageClassName: local-path
    - kind: clairpostgres
      managed: true
      overrides:
        volumeSize: 70Gi
        storageClassName: local-path

where:

spec.components.clair.overrides.volumeSize

Specifies the desired capacity for the ephemeral scratch volume.

spec.components.clair.overrides.storageClassName

Specifies the storage class to use for the ephemeral scratch volume.

spec.components.postgres.overrides.volumeSize

Specifies the desired capacity for the database persistent volume.

spec.components.postgres.overrides.storageClassName

Specifies the storage class to use for the database persistent volume.

spec.components.clairpostgres.overrides.volumeSize

Specifies the desired capacity for the database persistent volume.

spec.components.clairpostgres.overrides.storageClassName

Specifies the storage class to use for the database persistent volume.

Configure ephemeral storage for managed Clair

To increase Clair scratch space for image layer extraction on Project Quay, you can set volumeSize and storageClassName overrides on the managed clair component in the QuayRegistry custom resource.

When you deploy managed Clair, the Operator provisions an ephemeral volume named indexer-layer-storage for temporary image layer extraction during vulnerability scanning. If your registry scans large images, this scratch space can fill and cause Clair pods to fail.

By default, the ephemeral volume requests 20Gi of storage and uses the cluster default storage class. You can override the capacity and storage class without adding new fields to the QuayRegistry CR. The Operator applies these values to the ephemeral volume mounted at /var/tmp.

Note

These overrides apply to Clair scratch storage at /var/tmp, not to the managed Clair PostgreSQL database. To resize the Clair database volume, configure overrides on the clairpostgres component.

Procedure
  1. Edit your QuayRegistry custom resource. For example:

    $ oc edit quayregistry <registry_name> -n <namespace>
  2. Under spec.components, add overrides.volumeSize and, optionally, overrides.storageClassName to the managed clair component. For example:

    spec:
      components:
        - kind: clair
          managed: true
          overrides:
            volumeSize: 50Gi
            storageClassName: fast-ssd

    When no overrides are specified, the Operator preserves the default 20Gi request and does not set a storage class.

  3. Save the changes and wait for the Operator to reconcile the Clair deployment.

  4. Verify that the Clair deployment includes the updated ephemeral volume claim template. Confirm that the storage and storageClassName values match your overrides:

    $ oc get deploy -n <namespace> <registry_name>-clair-app \
      -o jsonpath='{.spec.template.spec.volumes[?(@.name=="indexer-layer-storage")].ephemeral.volumeClaimTemplate.spec}{"\n"}'
    Example output
    {"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"fast-ssd","volumeMode":"Filesystem"}

TLS encryption for Operator-managed PostgreSQL

You can enable TLS encryption for Operator-managed PostgreSQL databases used by Project Quay and Clair. Configure TLS through the overrides.tls fields on the postgres and clairpostgres components in the QuayRegistry custom resource.

Important

This feature configures transport encryption for Operator-managed PostgreSQL connections. It is distinct from the managed tls component, which handles Project Quay’s external HTTPS endpoint.

When TLS is enabled, the Operator configures PostgreSQL to accept encrypted connections and mounts the TLS certificates on the database pod. TLS is opt-in. Existing deployments continue to use unencrypted database connections until you explicitly enable it.

Note

After you enable TLS, confirm that PostgreSQL reports ssl = on. Optionally verify that Project Quay or Clair client sessions appear in pg_stat_ssl, or that the registry database URI includes an sslmode setting such as verify-full, depending on your Operator version and configuration.

You can enable TLS independently for the postgres and clairpostgres components.

Certificate options

Table 3. Operator-managed PostgreSQL TLS certificate sources
Option Description

OpenShift Container Platform Service CA (default on OpenShift Container Platform)

When you set overrides.tls.enabled: true without a secretRef, the Operator uses the OpenShift Container Platform Service CA on clusters that support Routes. The Operator annotates the PostgreSQL Service with service.beta.openshift.io/serving-cert-secret-name and mounts the generated serving certificate.

Operator-generated self-signed certificates

On Kubernetes clusters without the OpenShift Container Platform Service CA, the Operator generates ECDSA P-256 certificates with a 10-year validity period when no secretRef is provided.

User-provided or cert-manager certificates

Reference a Secret containing ca.crt, tls.crt, and tls.key by using overrides.tls.secretRef.name. This format is compatible with cert-manager output. When the referenced Secret changes, the Operator reconciles the deployment.

Note

This feature does not provide mutual TLS (mTLS) or automatic certificate rotation. To integrate with enterprise PKI or automate rotation, use secretRef with cert-manager or your own certificate management process.

Enable TLS for Operator-managed PostgreSQL

To encrypt database traffic for Operator-managed PostgreSQL used by Project Quay and Clair, you can set overrides.tls.enabled: true on the postgres and clairpostgres components in the QuayRegistry custom resource.

Prerequisites
  • The postgres and/or clairpostgres components that you want to protect are set to managed: true.

  • You have access to edit the QuayRegistry custom resource in your registry namespace.

Procedure
  1. Edit your QuayRegistry custom resource. For example:

    $ oc edit quayregistry <registry_name> -n <namespace>
  2. Under spec.components, add overrides.tls.enabled: true to the managed PostgreSQL components. For example:

    spec:
      components:
        - kind: postgres
          managed: true
          overrides:
            tls:
              enabled: true
        - kind: clairpostgres
          managed: true
          overrides:
            tls:
              enabled: true
  3. Save the changes and wait for the Operator to reconcile the registry.

    Note

    On OpenShift Container Platform, the Operator might briefly report RolloutBlocked while it waits for the Service CA to create the serving certificate Secret. Reconciliation retries until the Secret is available.

  4. Verify that PostgreSQL has SSL enabled:

    $ oc exec -n <namespace> deploy/<registry_name>-quay-database -c postgres -- \
      bash -lc 'psql -U "$POSTGRESQL_USER" -d "$POSTGRESQL_DATABASE" -tAc "SHOW ssl;"'

    The command should return on. For Clair, run the same check against the Clair PostgreSQL deployment when that component is managed.

  5. Verify that the QuayRegistry status shows a healthy deployment and that RolloutBlocked is not stuck in an error state.

  6. Optional: Confirm client TLS sessions with the following command:

    $ oc exec -n <namespace> deploy/<registry_name>-quay-database -c postgres -- \
      bash -lc 'psql -U "$POSTGRESQL_USER" -d "$POSTGRESQL_DATABASE" -tAc "SELECT count(*) FROM pg_stat_ssl s JOIN pg_stat_activity a ON s.pid = a.pid WHERE s.ssl = true AND a.client_addr IS NOT NULL;"'

    A count greater than 0 indicates active TLS client connections. If the count remains 0, PostgreSQL still has TLS enabled on the server, but clients might not yet be connecting with SSL. Check the registry database URI or connection arguments for an sslmode value such as verify-full, and review Operator Events for certificate or CA errors.

Provide custom TLS certificates

To use your own certificates instead of Operator-generated or Service CA certificates, create a Secret and reference it from the component override.

Prerequisites
  • TLS is enabled on the target component (overrides.tls.enabled: true).

  • You have a Secret containing PEM-encoded ca.crt, tls.crt, and tls.key entries.

  • Your TLS certificate includes Subject Alternative Name (SAN) entries that match the PostgreSQL Service DNS names. For the postgres component, include the following names:

    <registry_name>-quay-database
    <registry_name>-quay-database.<namespace>
    <registry_name>-quay-database.<namespace>.svc
    <registry_name>-quay-database.<namespace>.svc.cluster.local
    localhost

    For the clairpostgres component, include the following names:

    <registry_name>-clair-postgres
    <registry_name>-clair-postgres.<namespace>
    <registry_name>-clair-postgres.<namespace>.svc
    <registry_name>-clair-postgres.<namespace>.svc.cluster.local
    localhost
Procedure
  1. Create a Secret in the same namespace as your QuayRegistry custom resource. For example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: <postgres_or_clairpostgres_name>-tls
      namespace: <namespace>
    type: Opaque
    stringData:
      ca.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      tls.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      tls.key: |
        -----BEGIN EC PRIVATE KEY-----
        ...
        -----END EC PRIVATE KEY-----
  2. Reference the Secret from the PostgreSQL component override:

        - kind: postgres
          managed: true
          overrides:
            tls:
              enabled: true
              secretRef:
                name: <postgres_or_clairpostgres_name>-tls
  3. Save the QuayRegistry and wait for reconciliation.

Verification
  1. Connect to the Project Quay PostgreSQL pod and check that SSL is enabled:

    $ oc exec -n <namespace> deploy/<registry>-quay-database -c postgres -- bash -lc 'psql -U "$POSTGRESQL_USER" -d "$POSTGRESQL_DATABASE" -tAc "SHOW ssl;"'

    The command should return on.

  2. Optional: Confirm that client connections use SSL:

    $ oc exec -n <namespace> deploy/<registry_name>-quay-database -c postgres -- \
      bash -lc 'psql -U "$POSTGRESQL_USER" -d "$POSTGRESQL_DATABASE" -tAc "SELECT count(*) FROM pg_stat_ssl s JOIN pg_stat_activity a ON s.pid = a.pid WHERE s.ssl = true AND a.client_addr IS NOT NULL;"'

    A count greater than 0 indicates active TLS client connections. If the count remains 0, PostgreSQL still has TLS enabled on the server, but clients might not yet be connecting with SSL. Check the registry database URI or connection arguments for an sslmode value such as verify-full, and review Operator Events for certificate or CA errors.

Disable TLS

You can disable TLS on an existing deployment without data loss by removing or setting overrides.tls.enabled to false on the affected components. The Operator removes TLS configuration from PostgreSQL and restores unencrypted connection strings.

Procedure
  1. Edit the QuayRegistry custom resource.

  2. Set overrides.tls.enabled: false or remove the overrides.tls block from the postgres and/or clairpostgres components.

  3. Save the changes and wait for the Operator to reconcile the registry.

Troubleshooting Operator-managed PostgreSQL TLS configurations

If TLS configuration is invalid, the Operator sets a RolloutBlocked condition on the QuayRegistry and emits Kubernetes Events with details.

Common causes include:

  • A referenced TLS Secret does not exist

  • The Secret is missing ca.crt, tls.crt, or tls.key

  • The certificate and private key do not match

  • The certificate has expired

On OpenShift Container Platform, if the Service CA serving certificate Secret has not yet been created, the Operator emits Events such as a PostgreSQL service CA error and retries until the Secret is available. A brief RolloutBlocked condition during that wait is expected.

If SHOW ssl; returns on but pg_stat_ssl shows no TLS client sessions, PostgreSQL is accepting encrypted connections, but Project Quay or Clair might still be connecting without SSL. Check the registry configuration for an sslmode value on the database URI or connection arguments, and review Operator Events for certificate or CA problems.

For external PostgreSQL databases, continue to configure TLS through DB_URI and related configuration fields in the config bundle. For more information, see Database configuration fields.

Container Security Operator

The Container Security Operator (CSO) scans images for active pods and reports vulnerabilities through the Kubernetes API. You can use CSO with Project Quay and Clair to monitor security issues in running workloads.

Important

The Container Security Operator has been deprecated and planned for removal in a future release of Project Quay and OpenShift Container Platform. The official replacement product of the Container Security Operator is Red Hat Advanced Cluster Security for Kubernetes.

Note

The CSO does not work without Project Quay and Clair.

The Container Security Operator (CSO) includes the following features:

  • Watches containers associated with pods on either specified or all namespaces.

  • Queries the container registry where the containers came from for vulnerability information, provided that an image’s registry supports image scanning, such as a Project Quay registry with Clair scanning.

  • Exposes vulnerabilities through the ImageManifestVuln object in the Kubernetes API.

Note

To see instructions on installing the CSO on Kubernetes, select the Install button on the Container Security Operator OperatorHub page.

Download and run the Container Security Operator in OpenShift Container Platform

To scan container images for vulnerabilities in your OpenShift Container Platform cluster, you can install the Container Security Operator from the OpenShift Container Platform OperatorHub and view vulnerability information in the dashboard.

The following procedure sets up the Container Security Operator to monitor pods and expose vulnerability data through the Kubernetes API.

Note

In the following procedure, the Container Security Operator is installed in the marketplace-operators namespace. This allows the CSO to be used in all namespaces of your OpenShift Container Platform cluster.

After completing this procedure, you are made aware of what images are vulnerable, what you must do to fix those vulnerabilities, and every namespace that the image was run in. Knowing this, you can perform the following actions:

  • Alert users who are running the image that they need to correct the vulnerability.

  • Stop the images from running by deleting the deployment or the object that started the pod that the image is in.

Procedure
  1. On the OpenShift Container Platform console page, select Operators → OperatorHub and search for Container Security Operator.

  2. Select the Container Security Operator, then select Install to go to the Create Operator Subscription page.

  3. Check the settings (all namespaces and automatic approval strategy, by default), and select Subscribe. The Container Security appears after a few moments on the Installed Operators screen.

  4. Optional: you can add custom certificates to the CSO. In this example, create a certificate named quay.crt in the current directory. Then, run the following command to add the certificate to the CSO:

    $ oc create secret generic container-security-operator-extra-certs --from-file=quay.crt -n openshift-operators
    Note

    You must restart the Operator pod for the new certificates to take effect.

  5. Navigate to Home → Overview. A link to Image Vulnerabilities appears under the status section, with a listing of the number of vulnerabilities found so far. Select the link to see a security breakdown, as shown in the following image:

    Access CSO scanning data from the OpenShift Container Platform dashboard

    Important

    The Container Security Operator currently provides broken links for Red Hat Security advisories. For example, the following link might be provided: https://access.redhat.com/errata/RHSA-2023:1842%20https://access.redhat.com/security/cve/CVE-2023-23916. The %20 in the URL represents a space character, however it currently results in the combination of the two URLs into one incomplete URL, for example, https://access.redhat.com/errata/RHSA-2023:1842 and https://access.redhat.com/security/cve/CVE-2023-23916. As a temporary workaround, you can copy each URL into your browser to navigate to the proper page. This is a known issue and will be fixed in a future version of Project Quay.

  6. You can do one of two things at this point to follow up on any detected vulnerabilities:

    1. Select the link to the vulnerability. You are taken to the container registry, Project Quay or other registry where the container came from, where you can see information about the vulnerability. The following figure shows an example of detected vulnerabilities from a Quay.io registry:

      The CSO points you to a registry containing the vulnerable image

    2. Select the namespaces link to go to the Image Manifest Vulnerabilities page, where you can see the name of the selected image and all namespaces where that image is running. The following figure indicates that a particular vulnerable image is running in two namespaces:

      View namespaces a vulnerable image is running in

Query image vulnerabilities from the CLI

To check for security vulnerabilities in your Project Quay container images, you can query vulnerability information from the command line using the oc get vuln command. You can also view detailed information about specific vulnerabilities by using the oc describe vuln command.

Procedure
  1. Enter the following command to query for detected vulnerabilities:

    $ oc get vuln --all-namespaces
    Example output
    NAMESPACE     NAME              AGE
    default       sha256.ca90...    6m56s
    skynet        sha256.ca90...    9m37s
  2. Optional. To display details for a particular vulnerability, identify a specific vulnerability and its namespace, and use the oc describe command. The following example shows an active container whose image includes an RPM package with a vulnerability:

    $ oc describe vuln --namespace <namespace> sha256.ac50e3752...
Name:         sha256.ac50e3752...
Namespace:    quay-enterprise
...
Spec:
  Features:
    Name:            nss-util
    Namespace Name:  centos:7
    Version:         3.44.0-3.el7
    Versionformat:   rpm
    Vulnerabilities:
      Description: Network Security Services (NSS) is a set of libraries...

Uninstall the Container Security Operator

To uninstall the Container Security Operator from your OpenShift Container Platform deployment, you must uninstall the Operator and delete the imagemanifestvulns.secscan.quay.redhat.com custom resource definition (CRD).

Without removing the CRD, image vulnerabilities are still reported on the OpenShift Container Platform Overview page.

Procedure
  1. On the OpenShift Container Platform web console, click Operators → Installed Operators.

  2. Click the menu kebab of the Container Security Operator.

  3. Click Uninstall Operator. Confirm your decision by clicking Uninstall in the popup window.

  4. Remove the imagemanifestvulns.secscan.quay.redhat.com custom resource definition by entering the following command:

    $ oc delete customresourcedefinition imagemanifestvulns.secscan.quay.redhat.com
    Example output
    customresourcedefinition.apiextensions.k8s.io "imagemanifestvulns.secscan.quay.redhat.com" deleted

Configuring AWS STS for Project Quay

You can configure AWS Security Token Service (STS) with Project Quay to authenticate to Amazon S3 by using temporary credentials. STS is available for standalone deployments, Red Hat Quay on OpenShift Container Platform, and Project Quay on Red Hat OpenShift Service on AWS (ROSA).

AWS STS provides temporary, limited-privilege credentials for trusted identities that assume an AWS Identity and Access Management (IAM) role. When Project Quay uses Amazon S3 as object storage, STS can provide temporary credentials for authorized S3 access.

You can configure AWS STS by using one of the following approaches:

Manual STS

Create an AWS IAM user and S3 role, and configure your Project Quay config.yaml file with the static IAM user credentials used to assume the role. This approach is supported for standalone, Red Hat Quay on OpenShift Container Platform, and ROSA deployments.

CCO-based STS (OpenShift Container Platform 4.14 or later)

On an AWS cluster with CCO in Manual mode, use a projected OpenShift Container Platform service account token and a CCO-generated AWS profile to assume an IAM role. The AWS SDK in Project Quay obtains and refreshes temporary credentials from AWS STS. This approach eliminates static IAM user credentials from the config bundle.

Create an IAM user

To configure AWS STS authentication for your Project Quay deployment, you can create an IAM user in the AWS console, copy the user ARN, and create access keys. This procedure sets up the IAM user that Project Quay uses to authenticate with Amazon S3 using temporary credentials.

Procedure
  1. Log in to the Amazon Web Services (AWS) console and navigate to the Identity and Access Management (IAM) console.

  2. In the navigation pane, under Access management click Users.

  3. Click Create User and enter the following information:

    1. Enter a valid username, for example, quay-user.

    2. For Permissions options, click Add user to group.

  4. On the review and create page, click Create user. You are redirected to the Users page.

  5. Click the username, for example, quay-user.

  6. Copy the ARN of the user, for example, arn:aws:iam::123456:user/quay-user.

  7. On the same page, click the Security credentials tab.

  8. Navigate to Access keys.

  9. Click Create access key.

  10. On the Access key best practices & alternatives page, click Command Line Interface (CLI), then, check the confirmation box. Then click Next.

  11. Optional. On the Set description tag - optional page, enter a description.

  12. Click Create access key.

  13. Copy and store the access key and the secret access key.

    Important

    This is the only time that the secret access key can be viewed or downloaded. You cannot recover it later. However, you can create a new access key any time.

  14. Click Done.

Create an S3 role

To enable AWS STS authentication for your Project Quay deployment, you can create an S3 role in the AWS IAM console with a custom trust policy that allows your IAM user to assume the role. This procedure sets up the role that grants S3 access permissions for temporary credential authentication.

Prerequisites
  • You have created an IAM user and stored the access key and the secret access key.

Procedure
  1. Navigate to the IAM dashboard.

  2. In the navigation pane, click Roles under Access management.

  3. Click Create role → Custom Trust Policy.

  4. Under the Principal configuration field, add your AWS ARN information. For example:

    {
        "Version": "2012-10-17",
        "Statement": [
       	 {
       		 "Sid": "Statement1",
       		 "Effect": "Allow",
       		 "Principal": {
       		 	"AWS": "arn:aws:iam::123456:user/quay-user"
       		 },
       		 "Action": "sts:AssumeRole"
       	 }
        ]
    }
  5. Click Next.

  6. On the Add permissions page, type AmazonS3FullAccess in the search box. Check the box to add that policy to the S3 role, then click Next.

  7. On the Name, review, and create page, enter the following information:

    1. Enter a role name, for example, example-role.

    2. Optional. Add a description.

  8. Click the Create role button. You are navigated to the Roles page. Under Role name, the newly created S3 should be available.

Configure Red Hat Quay on OpenShift Container Platform to use AWS STS

To configure your Red Hat Quay on OpenShift Container Platform deployment to use AWS STS for S3 authentication, you can edit the config.yaml file through the OpenShift Container Platform UI and update the DISTRIBUTED_STORAGE_CONFIG fields with your role ARN, bucket name, and access keys.

This procedure enables temporary credential authentication for object storage access.

Note

You can also edit and re-deploy your Red Hat Quay on OpenShift Container Platform config.yaml file directly instead of using the OpenShift Container Platform UI.

Prerequisites
  • You have configured a Role ARN.

  • You have generated a User Access Key.

  • You have generated a User Secret Key.

Procedure
  1. On the Home page of your OpenShift Container Platform deployment, click Operators → Installed Operators.

  2. Click Red Hat Quay.

  3. Click Quay Registry and then the name of your Project Quay registry.

  4. Under Config Bundle Secret, click the name of your registry configuration bundle, for example, quay-registry-config-bundle-qet56.

  5. On the configuration bundle page, click Actions to reveal a drop-down menu. Then click Edit Secret.

  6. Update your the DISTRIBUTED_STORAGE_CONFIG fields of your config.yaml file with the following information:

    # ...
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - STSS3Storage
        - sts_role_arn: <role_arn>
          s3_bucket: <s3_bucket_name>
          storage_path: <storage_path>
          s3_region: <region>
          sts_user_access_key: <s3_user_access_key>
          sts_user_secret_key: <s3_user_secret_key>
    # ...

    where:

    DISTRIBUTED_STORAGE_CONFIG.default.sts_role_arn

    Specifies the unique Amazon Resource Name (ARN) required when configuring AWS STS.

    DISTRIBUTED_STORAGE_CONFIG.default.s3_bucket

    Specifies the name of your s3 bucket.

    DISTRIBUTED_STORAGE_CONFIG.default.storage_path

    Specifies the storage path for data. Usually /datastorage.

    DISTRIBUTED_STORAGE_CONFIG.default.s3_region

    Specifies the Amazon Web Services region. Defaults to us-east-1.

    DISTRIBUTED_STORAGE_CONFIG.default.sts_user_access_key

    Specifies the generated AWS S3 user access key required when configuring AWS STS.

    DISTRIBUTED_STORAGE_CONFIG.default.sts_user_secret_key

    Specifies the generated AWS S3 user secret key required when configuring AWS STS.

  7. Click Save.

Verification
  1. Tag a sample image, for example, busybox, that will be pushed to the repository. For example:

    $ podman tag docker.io/library/busybox <quay-server.example.com>/<organization_name>/busybox:test
  2. Push the sample image by running the following command:

    $ podman push <quay-server.example.com>/<organization_name>/busybox:test
  3. Verify that the push was successful by navigating to the Organization that you pushed the image to in your Project Quay registry → Tags.

  4. Navigate to the Amazon Web Services (AWS) console and locate your s3 bucket.

  5. Click the name of your s3 bucket.

  6. On the Objects page, click datastorage/.

  7. On the datastorage/ page, the following resources should seen:

    • sha256/

    • uploads/

      These resources indicate that the push was successful, and that AWS STS is properly configured.

Configure Project Quay on Red Hat OpenShift Service on AWS to use AWS STS

To configure your Project Quay deployment on Red Hat OpenShift Service on AWS to use AWS STS for S3 authentication, you can update the IAM role trust policy to use federated identity, configure the config.yaml file, and annotate the service account with the role ARN.

This procedure enables web identity federation for temporary credential authentication.

Prerequisites
  • You have created an IAM user.

  • You have created an s3 Role ARN.

  • You have created a Custom Trust Policy that uses the Role ARN.

Procedure
  1. Get the serviceAccountIssuer resource by entering the following command:

    $ oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed -e "s/^https:\/\///"
    Example output
    oidc.op1.openshiftapps.com/123456
  2. On the Identity and Access Management (IAM) console of the Amazon Web Services (AWS) console:

    1. Click Roles.

    2. Click the name of the Role to be used with AWS STS, for example, example-role.

    3. Click the Trust relationships tab, which shows the JSON policy created during "Creating an S3 role". Update the JSON policy as follows:

      {
      	"Version": "2012-10-17",
      	"Statement": [
          	{
              	"Sid": "Statement1",
              	"Effect": "Allow",
              	"Principal": {
                  	"Federated": "arn:aws:iam::123456:oidc-provider/oidc.op1.openshiftapps.com/123456"
              	},
              	"Action": "sts:AssumeRoleWithWebIdentity",
              	"Condition": {
                  	"StringEquals": {
                      	"oidc.op1.openshiftapps.com/123456:sub": "system:serviceaccount:quay:registry-quay-app"
                  	}
              	}
          	}
      	]
      }

      where:

      Statement.Principal.Federated

      Specifies that the Principal parameter of the JSON policy is updated to Federated:<your_user_ARN>:<serviceAccountIssuer_domain_path>.

      Statement.Action

      Specifies that the Action parameter of the JSON policy is updated to sts:AssumeRoleWithWebIdentity.

      Statement.Condition

      Specifies that the Condition parameter of the JSON policy is updated to StringEquals”: “<serviceAccountIssuer>:sub”: “system:serviceAccount:<quay_namespace>:<quay_registry_using_serviceAccount>.

    4. Verify that your User ARN is configured correct, then click Next.

    5. On the Add permissions page, select AmazonS3FullAccess, then click Next.

    6. On the Name, review, and create page, provide your role a name, a description, verify your configuration, add any optional tags. Then, click Create Role.

  3. On the Roles page, click the new role and store the Role ARN resource. For example:

    arn:aws:iam::123456:role/test_s3_access
  4. On the Project Quay web console:

    1. Click Operators → Installed Operators.

    2. Click Red Hat Quay.

    3. Click Quay Registry and then the name of your Project Quay registry.

    4. Under Config Bundle Secret, click the name of your registry configuration bundle, for example, quay-registry-config-bundle-12345.

    5. On the configuration bundle page, click Actions to reveal a drop-down menu. Then click Edit Secret.

    6. Update your the DISTRIBUTED_STORAGE_CONFIG fields of your config.yaml file with the following information:

      # ...
      DISTRIBUTED_STORAGE_CONFIG:
         default:
          - STSS3Storage
            s3_bucket: <s3_bucket_name>
            storage_path: <storage_path>
            s3_region: <region>
      # ...

      where:

      DISTRIBUTED_STORAGE_CONFIG.default.s3_bucket

      Specifies the name of your s3 bucket.

      DISTRIBUTED_STORAGE_CONFIG.default.storage_path

      Specifies the storage path for data. Usually /datastorage.

      DISTRIBUTED_STORAGE_CONFIG.default.s3_region

      Specifies the Amazon Web Services region. Defaults to us-east-1.

  5. Click Save. Your QuayRegistry custom resource (CR) automatically restarts.

  6. Annotate the Service Account (SA) that executes pods with the EKS configuration values. For example:

    $ oc annotate sa registry-quay-app "eks.amazonaws.com/role-arn"="arn:aws:iam::123456:role/test_s3_access" "eks.amazonaws.com/audience"="sts.amazonaws.com" "eks.amazonaws.com/sts-regional-endpoints"="true"
Verification
  1. Tag a sample image, for example, busybox, that will be pushed to the repository. For example:

    $ podman tag docker.io/library/busybox <quay-server.example.com>/<organization_name>/busybox:test
  2. Push the sample image by running the following command:

    $ podman push <quay-server.example.com>/<organization_name>/busybox:test
  3. Verify that the push was successful by navigating to the Organization that you pushed the image to in your Project Quay registry → Tags.

  4. Navigate to the Amazon Web Services (AWS) console and locate your s3 bucket.

  5. Click the name of your s3 bucket.

  6. On the Objects page, click datastorage/.

  7. On the datastorage/ page, the following resources should seen:

    • sha256/

    • uploads/

      These resources indicate that the push was successful, and that AWS STS is properly configured.

About CCO-based AWS STS authentication for Red Hat Quay on OpenShift Container Platform

The standardized AWS Security Token Service (STS) configuration flow uses the OpenShift Container Platform Cloud Credential Operator (CCO) and the CredentialsRequest API to allow Project Quay workloads to authenticate to Amazon S3 without storing long-lived IAM access keys in the configBundleSecret resource.

When you set the ROLEARN environment variable in the Project Quay Operator Subscription, the Operator validates the cluster and storage configuration and creates a CredentialsRequest resource. CCO creates an AWS shared configuration profile that contains the role ARN and projected token-file path. The Operator validates and mounts this profile, together with a projected OpenShift Container Platform service account token, into the quay-app and quay-mirror pods.

The profile Secret does not contain an AWS access key, secret key, session token, or the projected service account token. The AWS SDK in Project Quay reads the profile and projected token, calls AWS STS AssumeRoleWithWebIdentity, and receives temporary AWS credentials. The AWS SDK refreshes the temporary credentials before they expire. The Project Quay Operator configures this identity flow but does not assume the IAM role or access S3.

This approach differs from the existing manual STS configuration in the following ways:

  • You do not need to create a static IAM user or generate access keys.

  • The Operator manages the CredentialsRequest lifecycle and validates the CCO-generated profile.

  • OpenShift Container Platform rotates the projected service account token, and the AWS SDK obtains and refreshes temporary AWS credentials.

  • The Operator reports credential provisioning and configuration errors through QuayRegistry status conditions.

Table 4. Comparison of STS approaches
Attribute Manual STS CCO-based STS

Identity source

Static IAM user access key and secret key

Projected OpenShift Container Platform service account token

AWS credentials

The configured IAM user credentials are used to assume a role

AWS STS returns temporary role credentials to the AWS SDK in Project Quay

Credential configuration

Stored in configBundleSecret as sts_user_access_key and sts_user_secret_key

CCO creates a profile Secret containing role_arn and web_identity_token_file; the token is projected separately

Operator involvement

Credentials are configured manually

The Operator creates and manages the CredentialsRequest and configures the quay-app and quay-mirror workloads

OpenShift Container Platform version required

Any supported OpenShift Container Platform version

OpenShift Container Platform 4.14 or later on AWS, with CCO in Manual mode and a configured service account issuer

Supported deployments

Standalone, Red Hat Quay on OpenShift Container Platform, and ROSA

AWS-based Red Hat Quay on OpenShift Container Platform deployments that meet the CCO and OIDC prerequisites

Prerequisites for CCO-based AWS STS authentication

To configure CCO-based AWS STS authentication for Red Hat Quay on OpenShift Container Platform, ensure that your cluster, AWS resources, and Project Quay configuration meet the listed prerequisites.

  • Your cluster runs OpenShift Container Platform 4.14 or later on AWS.

  • CCO is installed and its CloudCredential resource is configured with credentialsMode: Manual.

  • The CredentialsRequest API is available on the cluster.

  • The cluster has a non-empty service account issuer, and a corresponding IAM OIDC provider exists in the AWS account.

  • The IAM OIDC provider includes openshift in its client ID list.

  • You have an existing Amazon S3 bucket for Project Quay object storage.

  • You have an IAM role with a trust policy for the <quay_registry_name>-quay-app service account and a permission policy scoped to the S3 bucket.

  • The objectstorage component in the QuayRegistry CR is set to managed: false.

  • The config bundle contains a standard S3Storage configuration without s3_access_key or s3_secret_key.

Important

CCO Manual mode does not create the S3 bucket, IAM role, IAM permission policy, or AWS IAM OIDC provider. An AWS or cluster administrator must create these resources before enabling this feature.

One Project Quay Operator installation uses one ROLEARN value. If the Operator manages multiple QuayRegistry resources, the role trust policy must allow every intended <namespace>:<quay_registry_name>-quay-app service account subject, and the role permission policy must allow the required actions on every configured S3 bucket. Alternatively, use separately scoped Operator installations and roles.

The Operator reports STS provisioning and configuration problems by setting the existing RolloutBlocked condition on the QuayRegistry CR. The following table lists the associated condition reasons:

Condition reason Description

CredentialRequestPending

CCO has not provisioned the current CredentialsRequest generation or created the requested profile Secret. This condition is normally transient during initial setup or after the role ARN changes.

CredentialRequestNotProvisioned

Provisioning exceeded five minutes, or the CCO-generated profile is missing or invalid. Verify the current request generation, output Secret, role ARN, token path, and CCO logs.

ConflictingCredentials

ROLEARN is set, but an applicable S3Storage entry contains s3_access_key or s3_secret_key. The Operator blocks rollout until the static keys are removed or ROLEARN is removed.

ConfigInvalid

An AWS platform, CCO Manual mode, service account issuer, CredentialsRequest API, role ARN, or S3 configuration prerequisite is invalid. Correct the error reported in the condition message.

Create an IAM role for CCO-based AWS STS authentication

To use CCO-based STS authentication with Red Hat Quay on OpenShift Container Platform, create an AWS IAM role that trusts the Project Quay application service account through the cluster’s IAM OIDC provider and grants access to the required S3 bucket. The Project Quay workloads assume this role. The Project Quay Operator references the role ARN when it creates the CredentialsRequest, but does not assume the role or access S3.

Prerequisites
  • You have an existing S3 bucket for Project Quay object storage.

  • You know the QuayRegistry name and namespace.

  • You know the AWS account ID and AWS partition for your deployment. The standard partition is aws; AWS GovCloud uses aws-us-gov.

  • Your cluster has a service account issuer, and the corresponding IAM OIDC provider exists in the AWS account with openshift in its client ID list.

Important

The permission statements in the CredentialsRequest describe the permissions required by the workload. In CCO Manual mode, CCO does not create or update the IAM policy. Ensure that the policy attached to the existing role contains the required bucket-scoped actions.

Procedure
  1. Retrieve the cluster’s OIDC issuer URL:

    $ oc get authentication.config.openshift.io cluster \
      -o jsonpath='{.spec.serviceAccountIssuer}{"\n"}'
    Example output
    https://oidc.op1.openshiftapps.com/1234567890abcdef
  2. Remove the https:// prefix from the issuer URL to obtain the OIDC provider path used in the IAM provider ARN and trust-policy condition keys. For example:

    oidc.op1.openshiftapps.com/1234567890abcdef
  3. Log in to the AWS Identity and Access Management (IAM) console.

  4. Create a customer managed permission policy for the S3 bucket. Replace <aws_partition> and <s3_bucket_name> in the following policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation",
                    "s3:ListBucketMultipartUploads"
                ],
                "Resource": "arn:<aws_partition>:s3:::<s3_bucket_name>"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:DeleteObject",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload"
                ],
                "Resource": "arn:<aws_partition>:s3:::<s3_bucket_name>/*"
            }
        ]
    }
  5. In the IAM console navigation pane, click Roles → Create role.

  6. On the Select trusted entity page, select Web identity.

  7. From the Identity provider list, select the IAM OIDC provider that corresponds to the cluster’s service account issuer. If it is not available, configure the cluster’s IAM OIDC provider before continuing.

  8. From the Audience list, select openshift.

  9. Attach the customer managed S3 permission policy that you created.

  10. Enter a role name, for example, quay-sts-cco-role, and create the role.

  11. Open the new role, click the Trust relationships tab, and click Edit trust policy.

  12. Update the trust policy to restrict access to the Project Quay application service account. Replace the placeholder values in the following example:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:<aws_partition>:iam::<aws_account_id>:oidc-provider/<oidc_provider_path>"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "<oidc_provider_path>:aud": "openshift",
                        "<oidc_provider_path>:sub": "system:serviceaccount:<quay_namespace>:<quay_registry_name>-quay-app"
                    }
                }
            }
        ]
    }

    where:

    <aws_partition>

    Specifies the AWS partition, for example, aws or aws-us-gov.

    <aws_account_id>

    Specifies your 12-digit AWS account ID.

    <oidc_provider_path>

    Specifies the cluster OIDC issuer without the https:// prefix, for example, oidc.op1.openshiftapps.com/1234567890abcdef.

    <quay_namespace>

    Specifies the namespace of the QuayRegistry CR.

    <quay_registry_name>

    Specifies the name of the QuayRegistry CR. The Operator-created service account is <quay_registry_name>-quay-app.

  13. Save the trust policy.

  14. Copy the role ARN. For example:

    arn:aws:iam::123456789012:role/quay-sts-cco-role

    You use this ARN as the ROLEARN value when configuring the Project Quay Operator Subscription.

Configure Red Hat Quay on OpenShift Container Platform to use CCO-based AWS STS authentication

To enable CCO-based AWS STS authentication, configure unmanaged standard S3Storage without static AWS keys and set the ROLEARN environment variable in the Project Quay Operator Subscription.

Prerequisites
  • Your cluster and AWS resources meet the CCO-based AWS STS prerequisites.

  • You have an IAM role ARN with an OIDC trust policy for the <quay_registry_name>-quay-app service account and a permission policy scoped to the S3 bucket.

  • Your QuayRegistry CR has the objectstorage component set to managed: false.

  • You have an existing Amazon S3 bucket for Project Quay object storage.

Important

The S3Storage configuration must not contain s3_access_key or s3_secret_key when ROLEARN is set. If an applicable S3Storage entry contains either value, the Operator blocks rollout with the ConflictingCredentials reason.

Procedure
  1. Update the config.yaml file in your configBundleSecret to use standard S3Storage without static credentials. For example:

    DISTRIBUTED_STORAGE_CONFIG:
      default:
        - S3Storage
        - host: s3.amazonaws.com
          port: 443
          s3_bucket: <s3_bucket_name>
          storage_path: /datastorage/registry
          s3_region: <aws_region>
    DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - default
    DISTRIBUTED_STORAGE_PREFERENCE:
      - default

    where:

    DISTRIBUTED_STORAGE_CONFIG.default.s3_bucket

    Specifies the name of your S3 bucket.

    DISTRIBUTED_STORAGE_CONFIG.default.storage_path

    Specifies the storage path in the bucket.

    DISTRIBUTED_STORAGE_CONFIG.default.s3_region

    Specifies the AWS region of the S3 bucket, for example, us-east-1.

  2. Update the config bundle Secret with the edited config.yaml file. For example:

    $ oc create secret generic <config_bundle_secret_name> \
      --from-file=config.yaml \
      --namespace <quay_namespace> \
      --dry-run=client -o yaml | oc apply -f -
  3. Confirm that the QuayRegistry CR references the config bundle Secret and has unmanaged object storage:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: <quay_registry_name>
      namespace: <quay_namespace>
    spec:
      configBundleSecret: <config_bundle_secret_name>
      components:
        - kind: objectstorage
          managed: false
  4. Edit the Project Quay Operator Subscription:

    $ oc edit subscription quay-operator -n <quay_operator_namespace>
  5. Add the ROLEARN under spec.config.env:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: quay-operator
      namespace: <quay_operator_namespace>
    spec:
      config:
        env:
          - name: ROLEARN
            value: arn:<aws_partition>:iam::<aws_account_id>:role/<role_name>

    where:

    spec.config.env.name

    Must be set to ROLEARN, which identifies the IAM role that the Project Quay app and mirror workloads assume.

    spec.config.env.value

    The Operator uses this value when it creates the CredentialsRequest; the Operator does not assume the role.

  6. Save the Subscription. Operator Lifecycle Manager updates the Operator Deployment. During reconciliation, the Operator validates the AWS platform, CCO Manual mode, service account issuer, storage configuration, and CredentialsRequest API before creating the request.

Verification
  1. Verify that the Operator created the CredentialsRequest in the QuayRegistry namespace:

    $ oc get credentialsrequest <quay_registry_name>-aws-credentials \
      -n <quay_namespace>
  2. Verify that CCO provisioned the current request generation:

    $ oc get credentialsrequest <quay_registry_name>-aws-credentials \
      -n <quay_namespace> \
      -o jsonpath='{.status.provisioned}{"\t"}{.metadata.generation}{"\t"}{.status.lastSyncGeneration}{"\n"}'

    The provisioned value must be true. The lastSyncGeneration value must match the current metadata.generation value.

  3. Verify that CCO created the profile Secret by entering the following command:

    Important

    Do not print the Secret contents.

    $ oc get secret <quay_registry_name>-aws-sts-credentials \
      -n <quay_namespace>

    The Secret contains an AWS shared profile with the role ARN and projected token-file path. It does not contain temporary AWS credentials or the projected service account token.

  4. Check the QuayRegistry status conditions by entering the following command:

    $ oc get quayregistry <quay_registry_name> \
      -n <quay_namespace> \
      -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

    A healthy deployment does not report RolloutBlocked=True with a CredentialRequestPending, CredentialRequestNotProvisioned, ConflictingCredentials, or STS-related ConfigInvalid reason.

  5. Verify the AWS SDK environment configuration on the app and mirror Deployments by entering the following command:

    $ oc get deployment <quay_registry_name>-quay-app \
      -n <quay_namespace> \
      -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
      | grep '^AWS_'

    The output includes:

    AWS_SHARED_CREDENTIALS_FILE=/aws-sts/credentials
    AWS_SDK_LOAD_CONFIG=true

    Repeat this command for the <quay_registry_name>-quay-mirror Deployment when repository mirroring is enabled.

  6. Log in to Project Quay, push and pull a test image, and, if repository mirroring is enabled, run a mirror synchronization. Successful image operations confirm that the Project Quay workloads can exchange the projected token for temporary AWS credentials and access the S3 bucket.

    Note

    Do not print or copy the projected service account token into logs or support cases. The token is mounted separately at /var/run/secrets/openshift/serviceaccount/token and is rotated by OpenShift Container Platform. The AWS SDK in Project Quay obtains and refreshes the temporary AWS credentials.

Deploying IPv6 on Red Hat Quay on OpenShift Container Platform

IPv6 support lets you deploy Red Hat Quay on OpenShift Container Platform in IPv6-only environments such as Telco and Edge deployments. You can enable IPv6 by setting the FEATURE_LISTEN_IP_VERSION parameter to IPv6 in your config.yaml file.

For a list of known limitations, see IPv6 limitations.

Note

Currently, deploying IPv6 on the Red Hat Quay on OpenShift Container Platform is not supported on IBM Power and IBM Z.

Enable the IPv6 protocol family

To enable IPv6 support on your standalone Project Quay deployment, you can set FEATURE_LISTEN_IP_VERSION to IPv6 in your config.yaml file and restart the registry.

Warning

If your environment is configured for IPv4, but the FEATURE_LISTEN_IP_VERSION configuration field is set to IPv6, Project Quay fails to deploy.

Prerequisites
  • Your host and container software platform (Docker, Podman) must be configured to support IPv6.

Procedure
  1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to IPv6, for example:

    FEATURE_GOOGLE_LOGIN: false
    FEATURE_INVITE_ONLY_USER_CREATION: false
    FEATURE_LISTEN_IP_VERSION: IPv6
    FEATURE_MAILING: false
    FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
  2. Start, or restart, your Project Quay deployment.

  3. Check that your deployment is listening to IPv6 by entering the following command:

    $ curl <quay_endpoint>/health/instance
    Example output
    {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}
Results
  • After you enable IPv6 in your deployment’s config.yaml file, you can use all Project Quay features as usual when your environment is configured for IPv6 and is not affected by known IPv6 limitations.

IPv6 limitations

IPv6-only deployments cannot use certain Project Quay storage backends because those services lack IPv6 endpoints. Microsoft Azure Blob Storage and Amazon S3 CloudFront are not supported in IPv6 single-stack environments.

  • Currently, configuring your Project Quay deployment with the common Microsoft Azure Blob Storage configuration does not work on IPv6 single-stack environments. Because the endpoint of Microsoft Azure Blob Storage does not support IPv6, no workaround exists for this issue.

  • Currently, configuring your Project Quay deployment with Amazon S3 CloudFront does not work on IPv6 single-stack environments. Because the endpoint of Amazon S3 CloudFront does not support IPv6, no workaround exists for this issue.

Additional resources

Add custom SSL/TLS certificates when Project Quay is deployed on Kubernetes

To add custom SSL/TLS certificates to your Project Quay deployment on Kubernetes, you can base64 encode the certificate, add it to the config secret, and restart the pods.

The following procedure works around the limitation where the superuser panel certificate upload function does not work with Kubernetes deployments.

Prerequisites
  • Project Quay has been deployed.

  • You have a custom ca.crt file.

Procedure
  1. Base64 encode the contents of an SSL/TLS certificate by entering the following command:

    $ cat ca.crt | base64 -w 0
    Example output
    ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  2. Enter the following kubectl command to edit the quay-enterprise-config-secret file:

    $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret
  3. Add an entry for the certificate and paste the full base64 encoded stringer under the entry. For example:

      custom-cert.crt:
    c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  4. Use the kubectl delete command to remove all Project Quay pods. For example:

    $ kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6  quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms

    Afterwards, the Project Quay deployment automatically schedules replace pods with the new certificate data.