Optimize

Tune Operator component resources and autoscaling

Tune QuayRegistry component resources and Horizontal Pod Autoscaling by using the OpenShift web console or CLI.

About Horizontal Pod Autoscaling (HPA)

Horizontal Pod Autoscalers (HPAs) automatically adjust the number of running pods based on CPU and memory utilization. Project Quay deployments include managed HPAs for key components to ensure availability and performance during load spikes or maintenance events.

A typical Project Quay deployment includes the following pods:

Two pods for the Project Quay application (example-registry-quay-app-*)
One Redis pod for Project Quay logging (example-registry-quay-redis-*)
One PostgreSQL pod for metadata storage (example-registry-quay-database-*)
Two Quay mirroring pods (example-registry-quay-mirror-*)
Two pods for Clair (example-registry-clair-app-*)
One PostgreSQL pod for Clair (example-registry-clair-postgres-*)

HPAs are managed by default for the Quay, Clair, and Mirror components, each starting with two replicas to prevent downtime during upgrades, reconfigurations, or pod rescheduling ev

Managing Horizontal Pod Autoscaling

To customize scaling thresholds or replica limits for your Project Quay registry, you can set the horizontalpodautoscaler component to unmanaged in the QuayRegistry custom resource. You can then explicitly set replica counts for the quay, clair, and mirror components.

Note

The following procedure uses the OpenShift Container Platform web console to configure the Project Quay registry to use an external PostgreSQL database. For most users, use the web console is simpler.

This procedure can also be done by using the oc CLI and following the instructions in "Modifying the QuayRegistry CR by using the CLI" and " Modifying the configuration file by using the CLI".

Procedure

Edit your QuayRegistry CR:

$ oc edit quayregistry <quay_registry_name> -n <quay_namespace>

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: quay-registry
  namespace: quay-enterprise
spec:
  components:
    - kind: horizontalpodautoscaler
      managed: false
    - kind: quay
      managed: true
      overrides:
        replicas: null
    - kind: clair
      managed: true
      overrides:
        replicas: null
    - kind: mirror
      managed: true
      overrides:
        replicas: null
# ...

Create a custom HorizontalPodAutoscaler resource with your desired configuration, for example:

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2
metadata:
  name: quay-registry-quay-app
  namespace: quay-enterprise
spec:
  scaleTargetRef:
    kind: Deployment
    name: quay-registry-quay-app
    apiVersion: apps/v1
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 90
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 90

Apply the new HPA configuration to your cluster:

$ oc apply -f <custom_hpa>.yaml

horizontalpodautoscaler.autoscaling/quay-registry-quay-app created

Verification

Verify that your Project Quay application pods are running:

$ oc get pod | grep quay-app

quay-registry-quay-app-5b8fd49d6b-7wvbk         1/1     Running     0          34m
quay-registry-quay-app-5b8fd49d6b-jslq9         1/1     Running     0          3m42s
quay-registry-quay-app-5b8fd49d6b-pskpz         1/1     Running     0          43m

Verify that your custom HPA is active:

$ oc get hpa

NAME                     REFERENCE                           TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
quay-registry-quay-app   Deployment/quay-registry-quay-app   67%/90%, 54%/90%   3         20        3          51m

Configuring QuayRegistry CR resources

Configuring resources for managed components lets you adjust CPU and memory requests and limits for quay, clair, mirror, redis, and database pods. You can configure resources to run smaller test clusters or request more resources upfront to avoid performance issues.

The following components should not be set lower than their minimum requirements. Setting resources too low can cause issues with your deployment and, in some cases, result in failure of the pod’s deployment.

quay: Minimum of 6 GB, 2vCPUs
clair: Recommended of 2 GB memory, 2 vCPUs
clairpostgres: Minimum of 200 MB

You can configure resource requests on the OpenShift Container Platform UI or directly by updating the QuayRegistry CR via the CLI.

Important

The default values set for these components are the suggested values. Setting resource requests too high or too low might lead to inefficient resource utilization, or performance degradation, respectively.

Configuring resource requests by using the OpenShift Container Platform web console

To configure resource requests for your Project Quay registry components, you can use the OpenShift Container Platform web console to edit the QuayRegistry custom resource. You can set CPU and memory limits and requests for quay, clair, mirror, redis, and database pods.

Procedure

On the OpenShift Container Platform developer console, click Operators → Installed Operators → Red Hat Quay.
Click QuayRegistry.
Click the name of your registry, for example, example-registry.
Click YAML.

In the spec.components field, you can override the resources of all components by setting values for the .overrides.resources.limits and the overrides.resources.requests fields. You can also specify a storageClassName for postgres, clairpostgres, and clair resources. For postgres and clairpostgres, these fields must be defined during initial installation of the component. For clair, storageClassName applies to the ephemeral scratch volume used for image layer extraction. For example:

spec:
  components:
    - kind: clair
      managed: true
      overrides:
        resources:
          limits:
            cpu: "5"     # Limiting to 5 CPU (equivalent to 5000m or 5000 millicpu)
            memory: "18Gi"  # Limiting to 18 Gibibytes of memory
          requests:
            cpu: "4"     # Requesting 4 CPU
            memory: "4Gi"   # Requesting 4 Gibibytes of memory
    - kind: postgres
      managed: true
      overrides:
        storageClassName: "local-path"
        resources:
          limits: {}
          requests:
            cpu: "700m"   # Requesting 700 millicpu or 0.7 CPU
            memory: "4Gi"   # Requesting 4 Gibibytes of memory
    - kind: mirror
      managed: true
      overrides:
        resources:
          limits:
          requests:
            cpu: "800m"   # Requesting 800 millicpu or 0.8 CPU
            memory: "1Gi"   # Requesting 1 Gibibyte of memory
    - kind: quay
      managed: true
      overrides:
        resources:
          limits:
            cpu: "4"    # Limiting to 4 CPU
            memory: "10Gi"   # Limiting to 10 Gibibytes of memory
          requests:
            cpu: "4"   # Requesting 4 CPU
            memory: "10Gi"   # Requesting 10 Gibi of memory
    - kind: redis
      managed: true
      overrides:
        resources:
          limits:
            cpu: "400m"   # Limiting to 400 millicpu or 0.4 CPU
            memory: "400Mi"   # Limiting to 400 Mebibytes of memory
          requests:
            cpu: "400m"   # Requesting 400 millicpu or 0.4 CPU
            memory: "400Mi"   # Requesting 400 Mebibytes of memory
    - kind: clairpostgres
      managed: true
      overrides:
        storageClassName: "local-path"
        resources:
          limits:
            cpu: "800m"   # Limiting to 800 millicpu or 0.8 CPU
            memory: "3Gi"   # Limiting to 3 Gibibytes of memory
          requests: {}

limits: Setting the limits or requests fields to {} uses the default values for these resources.
limits: Leaving the limits or requests field empty puts no limitations on these resources.

Configuring resource requests by using the CLI

To configure resource requests for your Project Quay registry components after deployment, you can edit the QuayRegistry custom resource using the CLI. You can set CPU and memory limits and requests for quay, clair, mirror, redis, and database pods.

Procedure

Edit the QuayRegistry CR by entering the following command:
```
$ oc edit quayregistry <registry_name> -n <namespace>
```

Make any desired changes. For example:

    - kind: quay
      managed: true
      overrides:
        resources:
          limits: {}
          requests:
            cpu: "0.7"   # Requesting 0.7 CPU (equivalent to 500m or 500 millicpu)
            memory: "512Mi"   # Requesting 512 Mebibytes of memory

Save the changes.

Resize managed storage for Operator deployments

Resize managed storage for Red Hat Quay and Clair, apply volume size overrides, and configure ephemeral storage for managed Clair.

Resizing Managed Storage

To expand storage capacity for your Red Hat Quay on OpenShift Container Platform deployment, you can use the OpenShift Container Platform console to resize the PostgreSQL and Clair PostgreSQL persistent volume claims. This lets you increase storage beyond the default 50 GiB allocation when your registry needs more space.

When deploying Red Hat Quay on OpenShift Container Platform, three distinct persistent volume claims (PVCs) are deployed:

One for the PostgreSQL 15 registry.
One for the Clair PostgreSQL 15 registry.
One that uses NooBaa as a backend storage.

Note

The connection between Project Quay and NooBaa is done through the S3 API and ObjectBucketClaim API in OpenShift Container Platform. Project Quay leverages that API group to create a bucket in NooBaa, obtain access keys, and automatically set everything up. On the backend, or NooBaa, side, that bucket is creating inside of the backing store. As a result, NooBaa PVCs are not mounted or connected to Project Quay pods.

Prerequisites

You have cluster admin privileges on OpenShift Container Platform.

Procedure

Log into the OpenShift Container Platform console and select Storage → Persistent Volume Claims.
Select the desired PersistentVolumeClaim for either PostgreSQL 13 or Clair PostgreSQL 13, for example, example-registry-quay-postgres-13.
From the Action menu, select Expand PVC.
Enter the new size of the Persistent Volume Claim and select Expand.

After a few minutes, the expanded size should reflect in the PVC’s Capacity field.

Volume size overrides

Volume size overrides let you specify the desired capacity and storage class for managed components that use persistent or ephemeral volumes in your Project Quay deployment. You can set larger volumes upfront for performance reasons or when your storage backend does not support resizing.

The default size for the managed postgres and clairpostgres persistent volumes is 50Gi. The default size for the managed clair ephemeral scratch volume is 20Gi.

Use overrides.volumeSize and overrides.storageClassName on the postgres, clairpostgres, and clair components in the QuayRegistry custom resource. For postgres and clairpostgres, set these overrides during initial installation of the component. Changing storageClassName after a database PVC is created is not supported.

Note	To expand database storage on an existing deployment, see Resizing managed storage.

In the following example, storage overrides are applied to three managed components:

On clair, volumeSize and storageClassName configure ephemeral scratch storage for image layer extraction at /var/tmp.
On postgres and clairpostgres, volumeSize and storageClassName configure the managed PostgreSQL database persistent volumes.

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: quay-example
  namespace: quay-enterprise
spec:
  configBundleSecret: config-bundle-secret
  components:
    - kind: objectstorage
      managed: false
    - kind: route
      managed: true
    - kind: tls
      managed: false
    - kind: clair
      managed: true
      overrides:
        volumeSize: 50Gi
        storageClassName: fast-ssd
    - kind: postgres
      managed: true
      overrides:
        volumeSize: 70Gi
        storageClassName: local-path
    - kind: clairpostgres
      managed: true
      overrides:
        volumeSize: 70Gi
        storageClassName: local-path

where:

spec.components.clair.overrides.volumeSize: Specifies the desired capacity for the ephemeral scratch volume.
spec.components.clair.overrides.storageClassName: Specifies the storage class to use for the ephemeral scratch volume.
spec.components.postgres.overrides.volumeSize: Specifies the desired capacity for the database persistent volume.
spec.components.postgres.overrides.storageClassName: Specifies the storage class to use for the database persistent volume.
spec.components.clairpostgres.overrides.volumeSize: Specifies the desired capacity for the database persistent volume.
spec.components.clairpostgres.overrides.storageClassName: Specifies the storage class to use for the database persistent volume.

Configuring ephemeral storage for managed Clair

To increase Clair scratch space for image layer extraction on Project Quay, you can set volumeSize and storageClassName overrides on the managed clair component in the QuayRegistry custom resource.

When you deploy managed Clair, the Operator provisions an ephemeral volume named indexer-layer-storage for temporary image layer extraction during vulnerability scanning. If your registry scans large images, this scratch space can fill and cause Clair pods to fail.

By default, the ephemeral volume requests 20Gi of storage and uses the cluster default storage class. You can override the capacity and storage class without adding new fields to the QuayRegistry CR. The Operator applies these values to the ephemeral volume mounted at /var/tmp.

Note	These overrides apply to Clair scratch storage at `/var/tmp`, not to the managed Clair PostgreSQL database. To resize the Clair database volume, configure overrides on the `clairpostgres` component.

Procedure

Edit your QuayRegistry custom resource. For example:

$ oc edit quayregistry <registry_name> -n <namespace>

Under spec.components, add overrides.volumeSize and, optionally, overrides.storageClassName to the managed clair component. For example:
```
spec:
  components:
    - kind: clair
      managed: true
      overrides:
        volumeSize: 50Gi
        storageClassName: fast-ssd
```
When no overrides are specified, the Operator preserves the default 20Gi request and does not set a storage class.
Save the changes and wait for the Operator to reconcile the Clair deployment.

Verify that the Clair deployment includes the updated ephemeral volume claim template. Confirm that the storage and storageClassName values match your overrides:

$ oc get deploy -n <namespace> <registry_name>-clair-app \
  -o jsonpath='{.spec.template.spec.volumes[?(@.name=="indexer-layer-storage")].ephemeral.volumeClaimTemplate.spec}{"\n"}'

Example output

{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"fast-ssd","volumeMode":"Filesystem"}

Tune registry runtime performance

Tune rate limits, HTTP connections, worker processes, and large-artifact settings to match your traffic profile.

Rate limiting and performance configuration fields

The following configuration fields control rate limiting and performance-related behavior for your Project Quay deployment.

Table 1. Rate limiting and performance configuration fields
Field	Type	Description
FEATURE_RATE_LIMITS	Boolean	Whether to enable rate limits on API and registry endpoints. Setting FEATURE_RATE_LIMITS to `True` causes `nginx` to limit certain API calls to 30 per second. If that feature is not set, API calls are limited to 300 per second (effectively unlimited). Default: `False`
PROMETHEUS_NAMESPACE	String	The prefix applied to all exposed Prometheus metrics Default: `quay`

Rate limiting and performance example YAML

# ...
FEATURE_RATE_LIMITS: false
PROMETHEUS_NAMESPACE: quay
# ...

Environment variable configuration

The following environment variables tune Project Quay runtime behavior for geo-replication, database pooling, HTTP concurrency, and worker scaling.

Use environment variables cautiously. These options typically override or augment existing configuration mechanisms.

This section documents environment variables related to the following components:

Geo-replication preferences
Database connection pooling
HTTP connection concurrency
Worker process scaling

HTTP connection counts

Use worker connection count environment variables to limit simultaneous HTTP connections per Project Quay worker process globally or per component. These limits apply either globally or can be scoped to individual components (registry, web UI, or security scanning). By default, each worker process allows up to 50 parallel connections.

This setting is distinct from the number of worker processes.

These connection-related environment variables can be configured differently depending on your deployment type:

In standalone deployments, configure connection counts in the config.yaml file.
In Red Hat Quay on OpenShift Container Platform deployments, define the values in the env block of the QuayRegistry CR.

Table 2. HTTP connection count configuration variables
Variable	Type	Description
WORKER_CONNECTION_COUNT	Number	Global default for the maximum number of HTTP connections per worker process. Default: 50
WORKER_CONNECTION_COUNT_REGISTRY	Number	HTTP connections per registry worker. Default: `WORKER_CONNECTION_COUNT`
WORKER_CONNECTION_COUNT_WEB	Number	HTTP connections per web UI worker. Default: `WORKER_CONNECTION_COUNT`
WORKER_CONNECTION_COUNT_SECSCAN	Number	HTTP connections per Clair security scanner worker. Default: `WORKER_CONNECTION_COUNT`

HTTP connection configuration for standalone Project Quay deployments

WORKER_CONNECTION_COUNT: 10
WORKER_CONNECTION_COUNT_REGISTRY: 10
WORKER_CONNECTION_COUNT_WEB: 10
WORKER_CONNECTION_COUNT_SECSCAN: 10

HTTP connection configuration for Red Hat Quay on OpenShift Container Platform

env:
  - name: WORKER_CONNECTION_COUNT
    value: "10"
  - name: WORKER_CONNECTION_COUNT_REGISTRY
    value: "10"
  - name: WORKER_CONNECTION_COUNT_WEB
    value: "10"
  - name: WORKER_CONNECTION_COUNT_SECSCAN
    value: "10"

Worker process counts

Use worker count environment variables to set how many parallel processes handle registry, web UI, and security scanning requests in Project Quay.

If not explicitly set, Project Quay calculates the number of worker processes automatically based on the number of available CPU cores. While this dynamic scaling can optimize performance on larger machines, it might also lead to unnecessary resource usage in smaller environments.

In Red Hat Quay on OpenShift Container Platform deployments, the Operator sets the following default values:

WORKER_COUNT_REGISTRY: 8
WORKER_COUNT_WEB: 4
WORKER_COUNT_SECSCAN: 2

Table 3. Worker count variables
Variable	Type	Description
WORKER_COUNT	Number	Generic override for number of processes
WORKER_COUNT_REGISTRY	Number	Specifies the number of processes to handle Registry requests within the `Quay` container. Values: Integer between `8` and `64`
WORKER_COUNT_WEB	Number	Specifies the number of processes to handle UI/Web requests within the container. Values: Integer between `2` and `32`
WORKER_COUNT_SECSCAN	Number	Specifies the number of processes to handle Security Scanning (e.g. Clair) integration within the container. Values: Integer. Because the Operator specifies 2 vCPUs for resource requests and limits, setting this value between `2` and `4` is safe. However, users can run more, for example, `16`, if warranted.

Worker count configuration for standalone Project Quay deployments

WORKER_COUNT: 10
WORKER_COUNT_REGISTRY: 16
WORKER_COUNT_WEB: 8
WORKER_COUNT_SECSCAN: 4

Worker count configuration for Red Hat Quay on OpenShift Container Platform

env:
  - name: WORKER_COUNT
    value: "10"
  - name: WORKER_COUNT_REGISTRY
    value: "16"
  - name: WORKER_COUNT_WEB
    value: "8"
  - name: WORKER_COUNT_SECSCAN
    value: "4"

Preparing your registry to accept large artifacts

To accept large AI or ML artifacts in Project Quay, you can increase the minimum_chunk_size_mb value in your config.yaml file after you consult Red Hat Support.

Important

Before altering the minimum_chunk_size_mb configuration field, open a support case with Red Hat Support. Altering minimum_chunk_size_mb can have unintended consequences for your registry.

Altering this field can also slow down uploads. You should only alter this field if necessary.

Artificial intelligence (AI) or machine learning (ML) artifacts such as large-language models (LLMs), vector graphics, trained model files, or large datasets often require that Project Quay administrators modify their registry to suit the needs of pushing such larger artifacts. By default, Project Quay uses a minimum chunk size (or the pieces that a large file is split into during upload) of 5 MB. This means that larger layers, for example, 50 GB, result in 10,000 chunks. This can be confirmed based on the following formula:

50 GB = 50,000 MB
50,000 MB divided by Project Quay’s default minimum chunk size of 5 MB = 10,000 chunks

Some backend storage providers, for example, Amazon Web Services (AWS) S3, are unable to store artifacts larger than 50 GB because of a strict limitation of 10,000 parts per upload; attempting to push an artifact larger than 50 GB with Project Quay’s default of 5 MB results in S3 protocol violations.

As a workaround to this limitation, you can set the minimum_chunk_size_mb field in your config.yaml file to a value larger than 5 MB. For example:

# ...
minimum_chunk_size_mb: 20
# ...

Configuring minimum_chunk_size_mb to more than 5 MB allows your registry backend to accept artifacts larger than 50 GB and up to 200 GB. In the event that your artifact is larger than 200 GB, you could increase the minimum_chunk_size_mb value.

Consult Red Hat Support before you alter the minimum_chunk_size_mb configuration field.