Project Quay is an enterprise-quality container registry. Use Quay to build and store containers, then deploy them to the servers across your enterprise.

This procedure describes how to deploy a high availability, enterprise-quality Project Quay setup.

Important

The Project Quay configuration tool is unsupported for standalone high availability deployments. Create and maintain the config.yaml file manually, as described in this guide. For a full list of configuration fields, see Configure Project Quay.

Project Quay features

Project Quay is regularly released with new features and software updates. The following features are available for Project Quay deployments, however the list is not exhaustive:

  • High availability

  • Geo-replication

  • Repository mirroring

  • Docker v2, schema 2 (multi-arch) support

  • Continuous integration

  • Security scanning with Clair

  • Custom log rotation

  • Zero downtime garbage collection

  • 24/7 support

Users should check the Project Quay Release Notes for the latest feature information.

Project Quay support

Project Quay provides support for the following:

  • Multiple authentication and access methods

  • Multiple storage backends

  • Custom certificates for Quay, Clair, and storage backend containers

  • Application registries

  • Different container image types

Architecture

Project Quay includes several core components, both internal and external.

For a fuller architectural breakdown, see the Project Quay architecture guide.

Internal components

Project Quay includes the following internal components:

  • Quay (container registry). Runs the Quay container as a service, consisting of several components in the pod.

  • Clair. Scans container images for vulnerabilities and suggests fixes.

External components

Project Quay includes the following external components:

  • Database. Used by Project Quay as its primary metadata storage. Note that this is not for image storage.

  • Redis (key-value store). Stores live builder logs and the Project Quay tutorial. Also includes the locking mechanism that is required for garbage collection.

  • Cloud storage. For supported deployments, one of the following storage types must be used:

    • Public cloud storage. In public cloud environments, you should use the cloud provider’s object storage, such as Amazon Web Services’s Amazon S3 or Google Cloud’s Google Cloud Storage.

    • Private cloud storage. In private clouds, an S3 or Swift compliant Object Store is needed, such as Ceph RADOS, or OpenStack Swift.

Warning

Do not use "Locally mounted directory" Storage Engine for any production configurations. Mounted NFS volumes are not supported. Local storage is meant for Project Quay test-only installations.

Preparing for Project Quay (high availability)

Note

This procedure presents guidance on how to set up a highly available, production-quality deployment of Project Quay.

Prerequisites

Here are a few things you need to know before you begin the Project Quay high availability deployment:

  • Either Postgres or MySQL can be used to provide the database service. Postgres was chosen here as the database because it includes the features needed to support Clair security scanning. Other options include:

    • Crunchy Data PostgreSQL Operator: Although not supported directly by Red Hat, the Postgres Operator is available from Crunchy Data for use with Project Quay. If you take this route, you should have a support contract with Crunchy Data and work directly with them for usage guidance or issues relating to the operator and their database.

    • If your organization already has a high-availability (HA) database, you can use that database with Project Quay. See the Project Quay Support Policy for details on support for third-party databases and other components.

  • Ceph Object Gateway (also called RADOS Gateway) is one example of a product that can provide the object storage needed by Project Quay. If you want your Project Quay setup to do geo-replication, Ceph Object Gateway or other supported object storage is required. For cloud installations, you can use any of the following cloud object storage:

    • Amazon S3 (see S3 IAM Bucket Policy for details on configuring an S3 bucket policy for Quay)

    • Azure Blob Storage

    • Google Cloud Storage

    • Ceph Object Gateway

    • OpenStack Swift

    • CloudFront + S3

    • NooBaa S3 Storage

  • The haproxy server is used in this example, although you can use any proxy service that works for your environment.

  • Number of systems: This procedure uses seven systems (physical or virtual) that are assigned with the following tasks:

    • A: db01: Load balancer and database: Runs the haproxy load balancer and a Postgres database. Note that these components are not themselves highly available, but are used to indicate how you might set up your own load balancer or production database.

    • B: quay01, quay02, quay03: Quay and Redis: Three (or more) systems are assigned to run the Quay and Redis services.

    • C: ceph01, ceph02, ceph03, ceph04, ceph05: Ceph: Three (or more) systems provide the Ceph service, for storage. If you are deploying to a cloud, you can use the cloud storage features described earlier. This procedure employs an additional system for Ansible (ceph05) and one for a Ceph Object Gateway (ceph04).

Each system should have the following attributes:

  • Red Hat Enterprise Linux (RHEL) 8: Obtain the latest Red Hat Enterprise Linux 8 server media from the Downloads page and follow the installation instructions available in the Product Documentation for Red Hat Enterprise Linux 9.

    • Valid Red Hat Subscription: Configure a valid Red Hat Enterprise Linux 8 server subscription.

    • CPUs: Two or more virtual CPUs

    • RAM: 4GB for each A and B system; 8GB for each C system

    • Disk space: About 20GB of disk space for each A and B system (10GB for the operating system and 10GB for docker storage). At least 30GB of disk space for C systems (or more depending on required container storage).

Using podman

This document uses podman for creating and deploying containers. If you do not have podman available on your system, you should be able to use the equivalent docker commands. For more information on podman and related technologies, see Building, running, and managing Linux containers on Red Hat Enterprise Linux 8.

Note

Podman is strongly recommended for highly available, production quality deployments of Project Quay. Docker has not been tested with Project Quay 3.18, and will be deprecated in a future release.

Setting up the HAProxy load balancer and the PostgreSQL database

Use the following procedure to set up the HAProxy load balancer and the PostgreSQL database.

Prerequisites

  • You have installed the Podman or Docker CLI.

Procedure

  1. On the first two systems, q01 and q02, install the HAProxy load balancer and the PostgreSQL database. This configures HAProxy as the access point and load balancer for the following services running on other systems:

    • Project Quay (ports 80 and 443 on B systems)

    • Redis (port 6379 on B systems)

    • RADOS (port 7480 on C systems)

  1. Open all HAProxy ports in SELinux and selected HAProxy ports in the firewall:

    # setsebool -P haproxy_connect_any=on
# firewall-cmd --permanent --zone=public --add-port=6379/tcp --add-port=7480/tcp
success
# firewall-cmd --reload
success

  1. Configure the /etc/haproxy/haproxy.cfg to point to the systems and ports providing the Project Quay, Redis and Ceph RADOS services. The following are examples of defaults and added frontend and backend settings:

    #---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------

frontend  fe_http *:80
    default_backend             be_http
frontend  fe_https *:443
    default_backend             be_https
frontend fe_redis *:6379
   default_backend              be_redis
frontend  fe_rdgw *:7480
    default_backend             be_rdgw
backend be_http
    balance     roundrobin
    server quay01 quay01:80 check
    server quay02 quay02:80 check
    server quay03 quay03:80 check
backend be_https
    balance     roundrobin
    server quay01 quay01:443 check
    server quay02 quay02:443 check
    server quay03 quay03:443 check
backend be_rdgw
    balance     roundrobin
    server ceph01 ceph01:7480 check
    server ceph02 ceph02:7480 check
    server ceph03 ceph03:7480 check
backend be_redis
server quay01 quay01:6379 check inter 1s
server quay02 quay02:6379 check inter 1s
server quay03 quay03:6379 check inter 1s

    After the new haproxy.cfg file is in place, restart the HAProxy service by entering the following command:

    # systemctl restart haproxy

  2. Create a folder for the PostgreSQL database by entering the following command:

    $ mkdir -p /var/lib/pgsql/data

  3. Set the following permissions for the /var/lib/pgsql/data folder:

    $ chmod 777 /var/lib/pgsql/data

  4. Enter the following command to start the PostgreSQL database:

    $ sudo podman run -d --name postgresql_database \
    -v /var/lib/pgsql/data:/var/lib/pgsql/data:Z  \
    -e POSTGRESQL_USER=quayuser -e POSTGRESQL_PASSWORD=quaypass \
    -e POSTGRESQL_DATABASE=quaydb -p 5432:5432 \
    registry.redhat.io/rhel8/postgresql-13:1-109
    Note

    Data from the container will be stored on the host system in the /var/lib/pgsql/data directory.

  5. List the available extensions by entering the following command:

    $ sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_available_extensions" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
    Example output
       name    | default_version | installed_version |           comment
-----------+-----------------+-------------------+----------------------------------------
 adminpack | 1.0             |                   | administrative functions for PostgreSQL
...

  6. Create the pg_trgm extension by entering the following command:

    $ sudo podman exec -it postgresql_database /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm;" | /opt/rh/rh-postgresql96/root/usr/bin/psql -d quaydb'

  7. Confirm that the pg_trgm has been created by entering the following command:

    $ sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_extension" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
    Example output
     extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
---------+----------+--------------+----------------+------------+-----------+--------------
 plpgsql |       10 |           11 | f              | 1.0        |           |
 pg_trgm |       10 |         2200 | t              | 1.3        |           |
(2 rows)

  8. Alter the privileges of the Postgres user quayuser and grant them the superuser role to give the user unrestricted access to the database:

    $ sudo podman exec -it postgresql_database /bin/bash -c 'echo "ALTER USER quayuser WITH SUPERUSER;" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
    Example output
    ALTER ROLE

  9. If you have a firewalld service active on your system, run the following commands to make the PostgreSQL port available through the firewall:

    # firewall-cmd --permanent --zone=trusted --add-port=5432/tcp
    # firewall-cmd --reload

  10. Optional. If you do not have the postgres CLI package installed, install it by entering the following command:

    # yum install postgresql -y

  11. Use the psql command to test connectivity to the PostgreSQL database.

    Note

    To verify that you can access the service remotely, run the following command on a remote system.

    		 
    # psql -h localhost quaydb quayuser
    Example output
    Password for user test:
psql (9.2.23, server 9.6.5)
WARNING: psql version 9.2, server version 9.6.
         Some psql features might not work.
Type "help" for help.

test=> \q

Set Up Ceph

For this Project Quay configuration, we create a three-node Ceph cluster, with several other supporting nodes, as follows:

  • ceph01, ceph02, and ceph03 - Ceph Monitor, Ceph Manager and Ceph OSD nodes

  • ceph04 - Ceph RGW node

  • ceph05 - Ceph Ansible administration node

For details on installing Ceph nodes, see Installing Red Hat Ceph Storage on Red Hat Enterprise Linux.

Once you have set up the Ceph storage cluster, create a Ceph Object Gateway (also referred to as a RADOS gateway). See Installing the Ceph Object Gateway for details.

Install each Ceph node

On ceph01, ceph02, ceph03, ceph04, and ceph05, do the following:

  1. Review prerequisites for setting up Ceph nodes in Requirements for Installing Red Hat Ceph Storage. In particular:

  2. Prepare OSD storage (ceph01, ceph02, and ceph03 only). Set up the OSD storage on the three OSD nodes (ceph01, ceph02, and ceph03). See OSD Ansible Settings in Table 3.2 for details on supported storage types that you will enter into your Ansible configuration later. For this example, a single, unformatted block device (/dev/sdb), that is separate from the operating system, is configured on each of the OSD nodes. If you are installing on metal, you might want to add an extra hard drive to the machine for this purpose.

  3. Install Red Hat Enterprise Linux Server edition, as described in the RHEL 7 Installation Guide.

  4. Register and subscribe each Ceph node as described in the Registering Red Hat Ceph Storage Nodes. Here is how to subscribe to the necessary repos:

    # subscription-manager repos --disable=*
# subscription-manager repos --enable=rhel-7-server-rpms
# subscription-manager repos --enable=rhel-7-server-extras-rpms
# subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms
# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms
# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms

  5. Create an ansible user with root privilege on each node. Choose any name you like. For example:

    # USER_NAME=ansibleadmin
# useradd $USER_NAME -c "Ansible administrator"
# passwd $USER_NAME
New password: *********
Retype new password: *********
# cat << EOF >/etc/sudoers.d/admin
admin ALL = (root) NOPASSWD:ALL
EOF
# chmod 0440 /etc/sudoers.d/$USER_NAME

Configure the Ceph Ansible node (ceph05)

Log into the Ceph Ansible node (ceph05) and configure it as follows. You will need the ceph01, ceph02, and ceph03 nodes to be running to complete these steps.

  1. In the Ansible user’s home directory create a directory to store temporary values created from the ceph-ansible playbook

    # USER_NAME=ansibleadmin
# sudo su - $USER_NAME
[ansibleadmin@ceph05 ~]$ mkdir ~/ceph-ansible-keys

  2. Enable password-less ssh for the ansible user. Run ssh-keygen on ceph05 (leave passphrase empty), then run and repeat ssh-copy-id to copy the public key to the Ansible user on ceph01, ceph02, and ceph03 systems:

    # USER_NAME=ansibleadmin
# sudo su - $USER_NAME
[ansibleadmin@ceph05 ~]$ ssh-keygen
[ansibleadmin@ceph05 ~]$ ssh-copy-id $USER_NAME@ceph01
[ansibleadmin@ceph05 ~]$ ssh-copy-id $USER_NAME@ceph02
[ansibleadmin@ceph05 ~]$ ssh-copy-id $USER_NAME@ceph03
[ansibleadmin@ceph05 ~]$ exit
#

  3. Install the ceph-ansible package:

    # yum install ceph-ansible

  4. Create a symbolic between these two directories:

    # ln -s /usr/share/ceph-ansible/group_vars \
    /etc/ansible/group_vars

  5. Create copies of Ceph sample yml files to modify:

    # cd /usr/share/ceph-ansible
# cp group_vars/all.yml.sample group_vars/all.yml
# cp group_vars/osds.yml.sample group_vars/osds.yml
# cp site.yml.sample site.yml

  6. Edit the copied group_vars/all.yml file. See General Ansible Settings in Table 3.1 for details. For example:

    ceph_origin: repository
ceph_repository: rhcs
ceph_repository_type: cdn
ceph_rhcs_version: 3
monitor_interface: eth0
public_network: 192.168.122.0/24

    Note that your network device and address range may differ.

  7. Edit the copied group_vars/osds.yml file. See the OSD Ansible Settings in Table 3.2 for details. In this example, the second disk device (/dev/sdb) on each OSD node is used for both data and journal storage:

    osd_scenario: collocated
devices:
  - /dev/sdb
dmcrypt: true
osd_auto_discovery: false

  8. Edit the /etc/ansible/hosts inventory file to identify the Ceph nodes as Ceph monitor, OSD and manager nodes. In this example, the storage devices are identified on each node as well:

    [mons]
ceph01
ceph02
ceph03

[osds]
ceph01 devices="[ '/dev/sdb' ]"
ceph02 devices="[ '/dev/sdb' ]"
ceph03 devices="[ '/dev/sdb' ]"

[mgrs]
ceph01 devices="[ '/dev/sdb' ]"
ceph02 devices="[ '/dev/sdb' ]"
ceph03 devices="[ '/dev/sdb' ]"

  9. Add this line to the /etc/ansible/ansible.cfg file, to save the output from each Ansible playbook run into your Ansible user’s home directory:

    retry_files_save_path = ~/

  10. Check that Ansible can reach all the Ceph nodes you configured as your Ansible user:

    # USER_NAME=ansibleadmin
# sudo su - $USER_NAME
[ansibleadmin@ceph05 ~]$ ansible all -m ping
ceph01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
ceph02 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
ceph03 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
[ansibleadmin@ceph05 ~]$

  11. Run the ceph-ansible playbook (as your Ansible user):

    [ansibleadmin@ceph05 ~]$ cd /usr/share/ceph-ansible/
[ansibleadmin@ceph05 ~]$ ansible-playbook site.yml

    At this point, the Ansible playbook will check your Ceph nodes and configure them for the services you requested. If anything fails, make needed corrections and rerun the command.

  12. Log into one of the three Ceph nodes (ceph01, ceph02, or ceph03) and check the health of the Ceph cluster:

    # ceph health
HEALTH_OK

  13. On the same node, verify that monitoring is working using rados:

    # ceph osd pool create test 8
# echo 'Hello World!' > hello-world.txt
# rados --pool test put hello-world hello-world.txt
# rados --pool test get hello-world fetch.txt
# cat fetch.txt
Hello World!

Install the Ceph Object Gateway

On the Ansible system (ceph05), configure a Ceph Object Gateway to your Ceph Storage cluster (which will ultimately run on ceph04). See Installing the Ceph Object Gateway for details.

Set up Redis

With Red Hat Enterprise Linux 8 server installed on each of the three Project Quay systems (quay01, quay02, and quay03), install and start the Redis service as follows:

  1. Install / Deploy Redis: Run Redis as a container on each of the three quay0* systems:

    # mkdir -p /var/lib/redis
# chmod 777 /var/lib/redis
# sudo podman run -d -p 6379:6379 \
    -v /var/lib/redis:/var/lib/redis/data:Z \
    registry.redhat.io/rhel8/redis-5

  2. Check redis connectivity: You can use the telnet command to test connectivity to the redis service. Type MONITOR (to begin monitoring the service) and QUIT to exit:

    # yum install telnet -y
# telnet 192.168.122.99 6379
Trying 192.168.122.99...
Connected to 192.168.122.99.
Escape character is '^]'.
MONITOR
+OK
+1525703165.754099 [0 172.17.0.1:43848] "PING"
QUIT
+OK
Connection closed by foreign host.
Note

For more information on using podman and restarting containers, see the section "Using podman" earlier in this document.

Configuring Project Quay

Before you run the Project Quay service as a container, create the config.yaml file that defines database, Redis, storage, and registry settings for your high availability deployment.

Prerequisites

  • You completed the load balancer, database, Ceph, and Redis setup procedures in this guide.

  • You generated unique values for SECRET_KEY and DATABASE_SECRET_KEY. These values must remain consistent across all Project Quay nodes.

Procedure

  1. On the first Project Quay node, for example quay01, create a directory for the configuration bundle:

    # mkdir -p /mnt/quay/config

  2. Create a config.yaml file in that directory. The following example shows the minimum fields for a high availability deployment that uses the PostgreSQL database, Redis, and Ceph Object Gateway resources from earlier procedures:

    AUTHENTICATION_TYPE: Database
PREFERRED_URL_SCHEME: https
SERVER_HOSTNAME: quay.example.com (1)
SECRET_KEY: <secret_key_value>
DATABASE_SECRET_KEY: <database_secret_key_value>
SETUP_COMPLETE: true
DB_URI: postgresql://quayuser:quaypass@db01:5432/quaydb (2)
BUILDLOGS_REDIS:
  host: db01 (3)
  port: 6379
USER_EVENTS_REDIS:
  host: db01 (3)
  port: 6379
DISTRIBUTED_STORAGE_CONFIG:
  radosGWStorage:
    - RadosGWStorage
    - access_key: <access_key>
      bucket_name: <bucket_name>
      hostname: db01 (4)
      is_secure: false
      port: '7480'
      secret_key: <secret_key>
      storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_PREFERENCE:
  - radosGWStorage
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
SUPER_USERS:
  - quayadmin (5)

    1. Use the hostname or IP address that clients use to reach the Project Quay service, typically the load balancer front end.

    2. Match the database user, password, host, and database name from Setting up the HAProxy load balancer and the PostgreSQL database.

    3. Use the load balancer hostname that exposes Redis on port 6379.

    4. Use the load balancer hostname that exposes the Ceph RADOS gateway on port 7480. For additional RadosGW fields, see Ceph Object Gateway (RadosGW) storage example.

    5. Optional. Adds an initial superuser account. You can also add superusers after deployment by editing config.yaml.

  3. Configure additional registry settings as needed. The following fields are commonly updated for production high availability deployments:

  4. Copy the configuration directory, including config.yaml and any TLS certificate files, to each Project Quay node in the cluster, for example quay02 and quay03.

Deploying Project Quay

Deploy the Project Quay service on each node in your cluster by mounting the shared config.yaml bundle and starting the registry container.

For a basic setup, you can deploy on a single node. For high availability, deploy three or more nodes, for example quay01, quay02, and quay03.

Note

The resulting Project Quay service listens on port 8080 for HTTP and port 8443 for HTTPS. This differs from earlier releases, which listened on ports 80 and 443 respectively. This guide maps 8080 and 8443 to standard ports 80 and 443 on the host.
Procedure

  1. On each Project Quay node, confirm that /mnt/quay/config contains config.yaml and any required TLS certificate files from Configuring Project Quay.

  2. Deploy Project Quay on the node. After you authenticate to registry.redhat.io (see Accessing Project Quay), run the following command:

    Note

    Add -e DEBUGLOG=true to the podman run command line for the Quay container to enable debug-level logging. Add -e IGNORE_VALIDATION=true to bypass validation during the startup process.

    		 
    # sudo podman run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --privileged=true \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d quay.io/projectquay/quay:v3.18.0
    Note

    Create /mnt/quay/storage on each node before running the registry container. Image blobs are stored in Ceph Object Gateway for this high availability example, but Project Quay still uses /datastorage for the storage_path defined in config.yaml.

  3. Open a browser to the URL of the node running the Quay container.

  4. Log in to Project Quay with the superuser account defined in config.yaml and confirm that the registry is working.

  5. Repeat the deployment on the remaining Project Quay nodes in the cluster.

  6. Optional. To add Clair image scanning or repository mirroring, continue with the following sections.

Add Clair image scanning to Project Quay

Setting up and deploying Clair image scanning for your Project Quay deployment is described in Clair Security Scanning.

Add repository mirroring to Project Quay

Enabling repository mirroring allows you to create container image repositories on your Project Quay cluster that match the content of a selected external registry, then sync those repositories on a regular schedule and on demand.

Procedure

  1. Set FEATURE_REPO_MIRROR: true in your config.yaml file. For optional mirroring fields, see Enabling repository mirroring for Project Quay.

  2. Start the repository mirroring worker on a Project Quay node:

    $ sudo podman run -d --name mirroring-worker \
  -v /mnt/quay/config:/conf/stack:Z \
  -v /root/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt \
  quay.io/projectquay/quay:v3.18.0 repomirror

    If you do not use a custom CA certificate, omit the -v /root/ca.crt:…​ line.

  3. Restart the Project Quay registry containers so they load the updated configuration.

  4. Create mirrored repositories in the Project Quay web UI as described in Repository Mirroring in Project Quay.

Starting to use Project Quay

With Project Quay now running, you can:

  • Select Tutorial from the Quay home page to try the 15-minute tutorial. In the tutorial, you learn to log into Quay, start a container, create images, push repositories, view repositories, and change repository permissions with Quay.

  • Refer to the Use Project Quay for information on working with Project Quay repositories.

Upgrading a geo-replication deployment of standalone Project Quay

To upgrade a geo-replication deployment of standalone Project Quay, stop operations on all instances, back up the deployment, then follow the procedure to upgrade each system. Expect intermittent downtime when upgrading to the next y-stream release.

Important

  • When upgrading geo-replication Project Quay deployments to the next y-stream release (for example, Project Quay 3.7 → Project Quay 3.8), or geo-replication deployments, you must stop operations before upgrading.

  • There is intermittent downtime down upgrading from one y-stream release to the next.

  • It is highly recommended to back up your Project Quay deployment before upgrading.
Note

This procedure assumes that you are running Project Quay services on three (or more) systems. For more information, see Preparing for Project Quay high availability.
Prerequisites

  • You have logged into registry.redhat.io

Procedure

  1. Obtain a list of all Project Quay instances on each system running a Project Quay instance.

    1. Enter the following command on System A to reveal the Project Quay instances:

      $ sudo podman ps
      Example output
      CONTAINER ID  IMAGE                                      COMMAND         CREATED        STATUS            PORTS                                        NAMES
ec16ece208c0  registry.redhat.io/quay/quay-rhel8:v{producty-n1}  registry        6 minutes ago  Up 6 minutes ago  0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay01

    2. Enter the following command on System B to reveal the Project Quay instances:

      $ sudo podman ps
      Example output
      CONTAINER ID  IMAGE                                      COMMAND         CREATED        STATUS            PORTS                                        NAMES
7ae0c9a8b37d  registry.redhat.io/quay/quay-rhel8:v{producty-n1}  registry        5 minutes ago   Up 2 seconds ago   0.0.0.0:82->8080/tcp, 0.0.0.0:445->8443/tcp  quay02

    3. Enter the following command on System C to reveal the Project Quay instances:

      $ sudo podman ps
      Example output
      CONTAINER ID  IMAGE                                      COMMAND         CREATED        STATUS            PORTS                                        NAMES
e75c4aebfee9  registry.redhat.io/quay/quay-rhel8:v{producty-n1}  registry        4 seconds ago   Up 4 seconds ago   0.0.0.0:84->8080/tcp, 0.0.0.0:447->8443/tcp  quay03

  2. Temporarily shut down all Project Quay instances on each system.

    1. Enter the following command on System A to shut down the Project Quay instance:

      $ sudo podman stop ec16ece208c0

    2. Enter the following command on System B to shut down the Project Quay instance:

      $ sudo podman stop 7ae0c9a8b37d

    3. Enter the following command on System C to shut down the Project Quay instance:

      $ sudo podman stop e75c4aebfee9

  3. Obtain the latest Project Quay version, for example, Project Quay 3.18, on each system.

    1. Enter the following command on System A to obtain the latest Project Quay version:

      $ sudo podman pull registry.redhat.io/quay/quay-rhel8:{productminv}

    2. Enter the following command on System B to obtain the latest Project Quay version:

      $ sudo podman pull registry.redhat.io/quay/quay-rhel8:v{producty}

    3. Enter the following command on System C to obtain the latest Project Quay version:

      $ sudo podman pull registry.redhat.io/quay/quay-rhel8:{productminv}

  4. On System A of your highly available Project Quay deployment, run the new image version, for example, Project Quay 3.18:

    # sudo podman run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --name=quay01 \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d registry.redhat.io/quay/quay-rhel8:{productminv}

  5. Wait for the new Project Quay container to become fully operational on System A. You can check the status of the container by entering the following command:

    $ sudo podman ps
    Example output
    CONTAINER ID  IMAGE                                      COMMAND         CREATED        STATUS            PORTS                                        NAMES
70b9f38c3fb4  registry.redhat.io/quay/quay-rhel8:v{producty} registry        2 seconds ago   Up 2 seconds ago   0.0.0.0:82->8080/tcp, 0.0.0.0:445->8443/tcp  quay01

  6. Optional: Ensure that Project Quay is fully operation by navigating to the Project Quay UI.

  7. After ensuring that Project Quay on System A is fully operational, run the new image versions on System B and on System C.

    1. On System B of your highly available Project Quay deployment, run the new image version, for example, Project Quay 3.18:

      # sudo podman run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --name=quay02 \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d registry.redhat.io/quay/quay-rhel8:{productminv}

    2. On System C of your highly available Project Quay deployment, run the new image version, for example, Project Quay 3.18:

      # sudo podman run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --name=quay03 \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d registry.redhat.io/quay/quay-rhel8:{productminv}

  8. You can check the status of the containers on System B and on System C by entering the following command:

    $ sudo podman ps

Performing health checks on Project Quay deployments

To monitor the health of your Project Quay deployment and identify issues before they become critical, you can perform health checks. Health checks assess system functionality and provide information about the current state for troubleshooting.

Health checks help ensure that everything is working correctly, and can be used to identify potential issues before they become critical problems. By monitoring the health of a system, Project Quay administrators can address abnormalities or potential failures for things like geo-replication deployments, Operator deployments, standalone Project Quay deployments, object storage issues, and so on. Performing health checks can also help reduce the likelihood of encountering troubleshooting scenarios. Health check mechanisms can play a role in diagnosing issues by providing valuable information about the system’s current state. By comparing health check results with expected benchmarks or predefined thresholds, deviations or anomalies can be identified quicker.

Important

Links contained herein to any external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or its entities, products, or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Project Quay has several health check endpoints. The following table shows you the health check, a description, an endpoint, and an example output.

Table 1. Health check endpoints
Health check Description Endpoint Example output

instance

The instance endpoint acquires the entire status of the specific Project Quay instance. Returns a dict with key-value pairs for the following: auth, database, disk_space, registry_gunicorn, service_key, and web_gunicorn. Returns a number indicating the health check response of either 200, which indicates that the instance is healthy, or 503, which indicates an issue with your deployment.

https://{quay-ip-endpoint}/health/instance or https://{quay-ip-endpoint}/health

{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

endtoend

The endtoend endpoint conducts checks on all services of your Project Quay instance. Returns a dict with key-value pairs for the following: auth, database, redis, storage. Returns a number indicating the health check response of either 200, which indicates that the instance is healthy, or 503, which indicates an issue with your deployment.

https://{quay-ip-endpoint}/health/endtoend

{"data":{"services":{"auth":true,"database":true,"redis":true,"storage":true}},"status_code":200}

warning

The warning endpoint conducts a check on the warnings. Returns a dict with key-value pairs for the following: disk_space_warning. Returns a number indicating the health check response of either 200, which indicates that the instance is healthy, or 503, which indicates an issue with your deployment.

https://{quay-ip-endpoint}/health/warning

{"data":{"services":{"disk_space_warning":true}},"status_code":503}