Troubleshoot

Get help from Red Hat Support

Search the Red Hat Knowledgebase and submit a support case when you need help with Red Hat Quay.

Getting support

To get help with Project Quay or file a support ticket, you can use the Red Hat Customer Portal and Knowledgebase. You can search articles, submit a support case, or use the debugging tool and health endpoint to gather information before contacting support.

To identify issues with your deployment, you can use the Project Quay debugging tool, or check the health endpoint of your deployment to obtain information about your problem. After you have debugged or obtained health information about your deployment, you can search the Red Hat Knowledgebase for a solution or file a support ticket.

If you have a suggestion for improving this documentation or have found an error, submit a Jira issue to the ProjectQuay project. Provide specific details, such as the section name and Project Quay version.

About the Red Hat Knowledgebase

The Red Hat Knowledgebase provides rich content aimed at helping you make the most of Red Hat’s products and technologies. The Red Hat Knowledgebase consists of articles, product documentation, and videos outlining best practices on installing, configuring, and using Red Hat products. In addition, you can search for solutions to known issues, each providing concise root cause descriptions and remedial steps.

The Project Quay Support Team also maintains a consolidate troubleshooting article for Project Quay that details solutions to common problems. This evolving document can guide you through various issues effectively and efficiently.

Additional resources

Searching the Red Hat Knowledgebase

To find solutions for Project Quay issues, you can search the Red Hat Knowledgebase. Log in to the Customer Portal, enter keywords related to your problem, and apply the Project Quay and Knowledgebase filters.

Prerequisites

You have a Red Hat Customer Portal account.

Procedure

Log in to the Red Hat Customer Portal.
In the main Red Hat Customer Portal search field, input keywords and strings relating to the problem, including:
- Project Quay components (such as database)
- Related procedure (such as installation)
- Warnings, error messages, and other outputs related to explicit failures
Click Search.
Select the Project Quay product filter.
Select the Knowledgebase content type filter.

Submitting a support case

To file a support ticket for Project Quay, you can submit a support case from the Red Hat Customer Portal. Log in, open a support case, enter problem details, select Project Quay and your version, and optionally attach debug logs.

Prerequisites

You have a Red Hat Customer Portal account.
You have a Red Hat standard or premium Subscription.

Procedure

Log in to the Red Hat Customer Portal and select Open a support case.
Select the Troubleshoot tab.
For Summary, enter a concise but descriptive problem summary and further details about the symptoms being experienced, as well as your expectations.
Review the list of suggested Red Hat Knowledgebase solutions for a potential match against the problem that is being reported. If the suggested articles do not address the issue, continue to the following step.
For Product, select Project Quay.
Select the version of Project Quay that you are using.
Click Continue.
Optional. Drag and drop, paste, or browse to upload a file. This could be debug logs gathered from your Project Quay deployment.
Click Get support to file your ticket.

Enable debug mode to diagnose issues

Run standalone Red Hat Quay or the Operator in debug mode, including LDAP deployments, to capture detailed diagnostic logs.

Running Project Quay in debug mode

To gather debugging information for support cases and troubleshoot issues, you can run Project Quay in debug mode. Debug mode provides verbose logging that speeds up reproducing errors and supports root cause analysis for geo-replication, Operator, standalone deployments, and object storage.

Project Quay debug variables

Project Quay offers two configuration fields that can be added to your config.yaml file to help diagnose issues or help obtain log information.

Variable Type Description

DEBUGLOG

Boolean

Whether to enable or disable debug logs. Must be True or False.

USERS_DEBUG

Integer. Either 0 or 1.

Used to debug LDAP operations in clear text, including passwords. Must be used with DEBUGLOG=TRUE.

Important

Setting USERS_DEBUG=1 exposes credentials in clear text. This variable should be removed from the Project Quay deployment after debugging. The log file that is generated with this environment variable should be scrutinized, and passwords should be removed before sending to other users. Use with caution.

Running a standalone Project Quay deployment in debug mode

To run a standalone Project Quay deployment in debug mode and get verbose logging for troubleshooting, you can use podman run with DEBUGLOG=true. View debug output with the podman logs command.

Procedure

Enter the following command to run your standalone Project Quay deployment in debug mode:

$ podman run -p 443:8443 -p 80:8080 -e DEBUGLOG=true  -v /config:/conf/stack -v /storage:/datastorage -d {productrepo}/{quayimage}:{productminv}

To view the debug logs, enter the following command:
```
$ podman logs <quay_container_name>
```

Running an LDAP Project Quay deployment in debug mode

To run an LDAP Project Quay deployment in debug mode and debug LDAP operations, you can use podman run with DEBUGLOG=true and USERS_DEBUG=1. View debug output with the podman logs command.

Procedure

Enter the following command to run your LDAP Project Quay deployment in debug mode:

$ podman run -p 443:8443 -p 80:8080 -e DEBUGLOG=true -e USERS_DEBUG=1  -v /config:/conf/stack -v /storage:/datastorage -d {productrepo}/{quayimage}:{productminv}

To view the debug logs, enter the following command:

$ podman logs <quay_container_name>

Important

Running the Project Quay Operator in debug mode

To run the Project Quay Operator in debug mode and get verbose logging for troubleshooting, you can edit the QuayRegistry custom resource and add DEBUGLOG=true to the env overrides. After the Operator restarts, try pulling an image or dump logs from Quay pods for more information.

Procedure

Enter the following command to edit the QuayRegistry custom resource definition:
```
$ oc edit quayregistry <quay_registry_name> -n <quay_namespace>
```

Update the QuayRegistry to add the following parameters:

spec:
  - kind: quay
    managed: true
    overrides:
      env:
      - name: DEBUGLOG
        value: "true"

After the Project Quay Operator has restarted with debugging enabled, try pulling an image from the registry. If the pull is still slow, dump all logs from all Quay pods to a file, and check the files for more information.

Collect logs and configuration for troubleshooting

Collect Red Hat Quay logs, verbose output, and configuration details to diagnose failures or share evidence with support.

Logging information for Project Quay

To troubleshoot, monitor, and secure your Project Quay deployment, you can use log information from containers and pods. Logs support debugging, performance monitoring, security analysis, capacity planning, and deployment verification.

Some of the reasons why obtaining log information is valuable include the following:

Debugging and Troubleshooting: Logs provide insights into what’s happening inside the application, allowing developers and system administrators to identify and resolve issues. By analyzing log messages, one can identify errors, exceptions, warnings, or unexpected behavior that might occur during the application’s execution.
Performance Monitoring: Monitoring logs helps to track the performance of the application and its components. Monitoring metrics like response times, request rates, and resource utilization can help in optimizing and scaling the application to meet the demand.
Security Analysis: Logs can be essential in auditing and detecting potential security breaches. By analyzing logs, suspicious activities, unauthorized access attempts, or any abnormal behavior can be identified, helping in detecting and responding to security threats.
Tracking User Behavior: In some cases, logs can be used to track user activities and behavior. This is particularly important for applications that handle sensitive data, where tracking user actions can be useful for auditing and compliance purposes.
Capacity Planning: Log data can be used to understand resource utilization patterns, which can aid in capacity planning. By analyzing logs, one can identify peak usage periods, anticipate resource needs, and optimize infrastructure accordingly.
Error Analysis: When errors occur, logs can provide valuable context about what happened leading up to the error. This can help in understanding the root cause of the issue and facilitating the debugging process.
Verification of Deployment: Logging during the deployment process can help verify if the application is starting correctly and if all components are functioning as expected.
Continuous Integration/Continuous Deployment (CI/CD): In CI/CD pipelines, logging is essential to capture build and deployment statuses, allowing teams to monitor the success or failure of each stage.

Obtaining log information for Project Quay

To obtain log information for your Project Quay deployment and troubleshoot authentication, authorization, or object storage issues, you can use oc logs for Operator deployments or podman logs for standalone deployments. You can then search the Red Hat Knowledgebase or file a support ticket.

Procedure

If you are using the Project Quay Operator on OpenShift Container Platform, enter the following command to view the logs:
```
$ oc logs <quay_pod_name>
```

If you are on a standalone Project Quay deployment, enter the following command:

$ podman logs <quay_container_name>

Example output

...
gunicorn-web stdout | 2023-01-20 15:41:52,071 [205] [DEBUG] [app] Starting request: urn:request:0d88de25-03b0-4cf9-b8bc-87f1ac099429 (/oauth2/azure/callback) {'X-Forwarded-For': '174.91.79.124'}
...

Examining verbose logs

To get a detailed status check of your Project Quay database pod or container, you can use oc logs or podman logs with --previous and copy PostgreSQL logs with oc cp or podman cp. Enable DEBUGLOG=true for additional debugging information.

Note

Additional debugging information can be returned if you have deployed Project Quay in one of the following ways:

You have deployed Project Quay by passing in the DEBUGLOG=true variable.
You have deployed Project Quay with LDAP authentication enabled by passing in the DEBUGLOG=true and USERS_DEBUG=1 variables.
You have configured Red Hat Quay on OpenShift Container Platform by updating the QuayRegistry resource to include DEBUGLOG=true.

For more information, see "Running Project Quay in debug mode".

Procedure

Enter the following commands to examine verbose database logs.

If you are using the Project Quay Operator on OpenShift Container Platform, enter the following commands:

$ oc logs <quay_pod_name> --previous

$ oc logs <quay_pod_name> --previous -c <container_name>

$ oc cp <quay_pod_name>:/var/lib/pgsql/data/userdata/log/* /path/to/desired_directory_on_host

If you are using a standalone deployment of Project Quay, enter the following commands:

$ podman logs <quay_container_id> --previous

$ podman logs <quay_container_id> --previous -c <container_name>

$ podman cp <quay_container_id>:/var/lib/pgsql/data/userdata/log/* /path/to/desired_directory_on_host

Configuration information for Project Quay

To identify and resolve Project Quay configuration issues, you can check the configuration YAML. You can verify parameters, resource limits, connectivity, authentication, replication, and backup settings.

Checking the configuration YAML can help you address the following issues:

Incorrect Configuration Parameters: If the database is not functioning as expected or is experiencing performance issues, your configuration parameters could be at fault. By checking the configuration YAML, administrators can ensure that all the required parameters are set correctly and match the intended settings for the database.
Resource Limitations: The configuration YAML might specify resource limits for the database, such as memory and CPU limits. If the database is running into resource constraints or experiencing contention with other services, adjusting these limits can help optimize resource allocation and improve overall performance.
Connectivity Issues: Incorrect network settings in the configuration YAML can lead to connectivity problems between the application and the database. Ensuring that the correct network configurations are in place can resolve issues related to connectivity and communication.
Data Storage and Paths: The configuration YAML may include paths for storing data and logs. If the paths are misconfigured or inaccessible, the database may encounter errors while reading or writing data, leading to operational issues.
Authentication and Security: The configuration YAML may contain authentication settings, including usernames, passwords, and access controls. Verifying these settings is crucial for maintaining the security of the database and ensuring only authorized users have access.
Plugin and Extension Settings: Some databases support extensions or plugins that enhance functionality. Issues may arise if these plugins are misconfigured or not loaded correctly. Checking the configuration YAML can help identify any problems with plugin settings.
Replication and High Availability Settings: In clustered or replicated database setups, the configuration YAML may define replication settings and high availability configurations. Incorrect settings can lead to data inconsistency and system instability.
Backup and Recovery Options: The configuration YAML might include backup and recovery options, specifying how data backups are performed and how data can be recovered in case of failures. Validating these settings can ensure data safety and successful recovery processes.

By checking your configuration YAML, Project Quay administrators can detect and resolve these issues before they cause significant disruptions to the application or service relying on the database.

Obtaining database configuration information

To obtain database configuration information for your Project Quay deployment, you can use oc exec for Operator deployments or podman exec for standalone deployments to read postgresql.conf from the database pod or container.

Warning

Interacting with the PostgreSQL database is potentially destructive. Red Hat recommends that you perform the following procedure with a Project Quay Support Specialist.

Procedure

If you are using the Project Quay Operator on OpenShift Container Platform, enter the following command:
```
$ oc exec -it <database_pod> -- cat /var/lib/pgsql/data/userdata/postgresql.conf
```

If you are using a standalone deployment of Project Quay, enter the following command:

$ podman exec -it <database_container> cat /var/lib/pgsql/data/userdata/postgresql.conf

Additional resources

Reading the configuration file from an Operator deployment by using the OpenShift CLI

Troubleshoot Quay database and authentication issues

Troubleshoot Red Hat Quay database, authentication, and superuser password reset issues on standalone or Operator deployments.

Troubleshooting Project Quay components

To troubleshoot specific Project Quay components and resolve component-related issues, you can use the procedures in this document. You can also run in debug mode, obtain logs and config, perform health checks, then search the Red Hat Knowledgebase or file a support ticket.

By using the following procedures, you are able to troubleshoot common component issues. Afterwards, you can search for solutions on the Red Hat Knowledgebase or file a support ticket with the Red Hat Support team.

Troubleshooting the Project Quay database

To troubleshoot the Project Quay database and resolve connectivity, configuration, or resource issues, you can check deployment type, pod or container status, logs, connectivity, and configuration. You can also examine resource allocation and interact with the PostgreSQL database.

Image Metadata. The database stores metadata associated with container images, such as image names, versions, creation timestamps, and the user or organization that owns the image. This information allows for easy identification and organization of container images within the registry.
Image Tags. You can assign tags to container images for labeling and versioning. The PostgreSQL database maintains the mapping between image tags and their corresponding image manifests, so you can retrieve specific versions of container images based on the provided tags.
Image Layers. Container images are composed of multiple layers, which are stored as individual objects. The database records information about these layers, including their order, checksums, and sizes. This data is crucial for efficient storage and retrieval of container images.
User and Organization Data. Project Quay supports user and organization management, allowing users to authenticate and manage access to container images. The PostgreSQL database stores user and organization information, including usernames, email addresses, authentication tokens, and access permissions.
Repository Information. Project Quay organizes container images into repositories, which act as logical units for grouping related images. The database maintains repository data, including names, descriptions, visibility settings, and access control information, enabling users to manage and share their repositories effectively.
Event Logs. Project Quay tracks various events and activities related to image management and repository operations. These event logs, including image pushes, pulls, deletions, and repository modifications, are stored in the PostgreSQL database, providing an audit trail and allowing administrators to monitor and analyze system activities.

You can use the following procedures to troubleshoot the database:

Checking the type of deployment: Determine if the database is deployed as a container on a virtual machine or as a pod on OpenShift Container Platform.
Checking the container or pod status: Verify the status of the database pod or container using specific commands based on the deployment type.
Examining the database container or pod logs: Access and examine the logs of the database pod or container, including commands for different deployment types.
Checking the connectivity between Project Quay and the database pod: Check the connectivity between Project Quay and the database pod using relevant commands.
Checking the database configuration: Check the database configuration at various levels (OpenShift Container Platform or PostgreSQL level) based on the deployment type.
Checking resource allocation: Monitor resource allocation for the Project Quay deployment, including disk usage and other resource usage.
Interacting with the Project Quay database: Learn how to interact with the PostgreSQL database, including commands to access and query databases.

Troubleshooting Project Quay database issues

To troubleshoot Project Quay database issues and resolve common errors like database connection refused or authentication failed, you can check the database logs, verify credentials, and ensure the database is running.

Resetting superuser passwords on Project Quay standalone deployments

To reset the superuser password on your Project Quay standalone deployment, you can generate a bcrypt-hashed password with Python, exec into the PostgreSQL container with podman exec, and update the user table with the new hashed password.

Prerequisites

You have created a Project Quay superuser.
You have installed Python 3.9.
You have installed the pip package manager for Python.
You have installed the bcrypt package for pip.

Procedure

Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command:

$ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'

Example output

$2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm

Enter the following command to show the container ID of your Project Quay container registry:

$ sudo podman ps -a

Example output

CONTAINER ID  IMAGE                                       COMMAND         CREATED         STATUS             PORTS                   NAMES
70560beda7aa  registry.redhat.io/rhel8/redis-5:1         run-redis       2 hours ago     Up 2 hours ago     0.0.0.0:6379->6379/tcp                       redis
8012f4491d10  registry.redhat.io/quay/quay-rhel8:v3.8.2  registry        3 minutes ago   Up 8 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay
8b35b493ac05  registry.redhat.io/rhel8/postgresql-10:1   run-postgresql  39 seconds ago  Up 39 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay

Execute an interactive shell for the postgresql container image by entering the following command:
```
$ sudo podman exec -it 8b35b493ac05 /bin/bash
```
Re-enter the quay PostgreSQL database server, specifying the database, username, and host address:
```
bash-4.4$ psql -d quay -U quayuser -h 192.168.1.28 -W
```

Update the password_hash of the superuser admin who lost their password:

quay=> UPDATE public.user SET password_hash = '$2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm' where username = 'quayadmin';

Example output

UPDATE 1

Enter the following to command to ensure that the password_hash has been updated:

quay=> select * from public.user;

Example output

id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
-------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492

$ sudo podman login -u quayadmin -p newpass1234 http://quay-server.example.com --tls-verify=false

Example output

Login Succeeded!

Resetting superuser passwords on the Project Quay Operator

To reset the superuser password on your Project Quay Operator deployment, you can generate a bcrypt-hashed password with Python, log in to the PostgreSQL database with oc rsh and psql, and update the user table with the new hashed password.

Prerequisites

You have created a Project Quay superuser.
You have installed Python 3.9.
You have installed the pip package manager for Python.
You have installed the bcrypt package for pip.

Procedure

Log in to your Project Quay deployment.
On the OpenShift Container Platform UI, navigate to Workloads → Secrets.
Select the namespace for your Project Quay deployment, for example, Project quay.
Locate and store the PostgreSQL database credentials.

Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command:

$ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'

Example output

$2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y

On the CLI, log in to the database, for example:

$ oc rsh quayuser-quay-quay-database-669c8998f-v9qsl

Enter the following command to open a connection to the quay PostgreSQL database server, specifying the database, username, and host address:
```
sh-4.4$ psql -U quayuser-quay-quay-database -d quayuser-quay-quay-database -W
```
Enter the following command to connect to the default database for the current user:
```
quay=> \c
```

Update the password_hash of the superuser admin who lost their password:

quay=> UPDATE public.user SET password_hash = '$2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y' where username = 'quayadmin';

Enter the following to command to ensure that the password_hash has been updated:

quay=> select * from public.user;

Example output

id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
-------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492

Navigate to your Project Quay UI on OpenShift Container Platform and log in using the new credentials.

Troubleshooting Project Quay authentication

Authentication and authorization secure access to Project Quay and safeguard container images, verify identities, and enforce access controls.

The following authentication methods are supported by Project Quay:

Username and password. Users can authentication by providing their username and password, which are validated against the user database configured in Project Quay. This traditional method requires users to enter their credentials to gain access.
OAuth. Project Quay supports OAuth authentication, which allows users to authenticate using their credentials from third party services like Google, GitHub, or Keycloak. OAuth enables a seamless and federated login experience, eliminating the need for separate account creation and simplifying user management.
OIDC. OpenID Connect enables single sign-on (SSO) capabilities and integration with enterprise identity providers. With OpenID Connect, users can authenticate using their existing organizational credentials, providing a unified authentication experience across various systems and applications.
Token-based authentication. Users can obtain unique tokens that grant access to specific resources within Project Quay. Tokens can be obtained through various means, such as OAuth or by generating API tokens within the Project Quay user interface. Token-based authentication is often used for automated or programmatic access to the registry.
External identity provider. Project Quay can integrate with external identity providers, such as LDAP or AzureAD, for authentication purposes. This integration allows organizations to use their existing identity management infrastructure, enabling centralized user authentication and reducing the need for separate user databases.

Troubleshooting Project Quay authentication and authorization issues for specific users

To troubleshoot Project Quay authentication and authorization issues for specific users, you can exec into the Project Quay pod or container and query the federatedlogin and user tables to verify user records.

Procedure

Exec into the Project Quay pod or container. For more information, see "Interacting with the Project Quay database".

Enter the following command to show all users for external authentication:

quay=# select * from federatedlogin;

Example output

id | user_id | service_id |                service_ident                |               metadata_json
----+---------+------------+---------------------------------------------+-------------------------------------------
1 |       1 |          3 | testuser0                                   | {}
2 |       1 |          8 | PK7Zpg2Yu2AnfUKG15hKNXqOXirqUog6G-oE7OgzSWc | {"service_username": "live.com#testuser0"}
3 |       2 |          3 | testuser1                                    | {}
4 |       2 |          4 | 110875797246250333431                       | {"service_username": "testuser1"}
5 |       3 |          3 | testuser2                                    | {}
6 |       3 |          1 | 26310880                                    | {"service_username": "testuser2"}
(6 rows)

Verify that the users are inserted into the user table:

quay=# select username, email from "user";

Example output

username  |        email
-----------+----------------------
testuser0 | testuser0@outlook.com
testuser1  | testuser1@gmail.com
testuser2  | testuser2@redhat.com
(3 rows)

Troubleshoot Quay connectivity and storage issues

Troubleshoot Red Hat Quay connectivity, crashloop states, database pod access, resource allocation, and object storage issues.

Interacting with the Project Quay database

To interact with the Project Quay PostgreSQL database and troubleshoot authorization or authentication issues, you can exec into the database using oc exec for Red Hat Quay on OpenShift Container Platform or podman exec for standalone, then enter the PostgreSQL shell with psql.

Warning

Interacting with the PostgreSQL database is potentially destructive. Red Hat recommends that you perform the following procedure with a Project Quay Support Specialist.

Note	Interacting with the PostgreSQL database can also be used to troubleshoot authorization and authentication issues.

Procedure

Exec into the Project Quay database.
1. Enter the following commands to exec into the Project Quay database pod on OpenShift Container Platform:
  $ oc exec -it <quay_database_pod> -- psql
2. Enter the following command to exec into the Project Quay database on a standalone deployment:
  $ sudo podman exec -it <quay_container_name> /bin/bash
Enter the PostgreSQL shell.

Warning

Interacting with the PostgreSQL database is potentially destructive. Red Hat recommends that you perform the following procedure with a Project Quay Support Specialist.
1. If you are using the Project Quay Operator, enter the following command to enter the PostgreSQL shell:
  $ oc rsh <quay_pod_name> psql -U your_username -d your_database_name
2. If you are on a standalone Project Quay deployment, enter the following command to enter the PostgreSQL shell:
  bash-4.4$ psql -U your_username -d your_database_name

Troubleshooting crashloopbackoff states

To troubleshoot crashloopbackoff states for your Project Quay deployment and restore pods or containers, you can scale down the Quay Operator and database, then edit the database deployment as needed.

Procedure

If your container or pod is in a crashloopbackoff state, you can enter the following commands.

Enter the following command to scale down the Project Quay Operator:

$ oc scale deployment/quay-operator.v3.8.z  --replicas=0

Example output

deployment.apps/quay-operator.v3.8.z scaled

Enter the following command to scale down the Project Quay database:

$ oc scale deployment/<quay_database> --replicas=0

Example output

deployment.apps/<quay_database> scaled

Enter the following command to edit the Project Quay database:

Warning

Interacting with the PostgreSQL database is potentially destructive. Red Hat recommends that you perform the following procedure with a Project Quay Support Specialist.

$ oc edit deployment <quay_database>

...
 template:
   metadata:
     creationTimestamp: null
     labels:
       quay-component: <quay_database>
       quay-operator/quayregistry: quay-operator.v3.8.z
   spec:
     containers:
     - env:
       - name: POSTGRESQL_USER
         value: postgres
       - name: POSTGRESQL_DATABASE
         value: postgres
       - name: POSTGRESQL_PASSWORD
         value: postgres
       - name: POSTGRESQL_ADMIN_PASSWORD
         value: postgres
       - name: POSTGRESQL_MAX_CONNECTIONS
         value: "1000"
       image: registry.redhat.io/rhel8/postgresql-10@sha256:a52ad402458ec8ef3f275972c6ebed05ad64398f884404b9bb8e3010c5c95291
       imagePullPolicy: IfNotPresent
       name: postgres
       command: ["/bin/bash", "-c", "sleep 86400"]
...

Add the command: ["/bin/bash", "-c", "sleep 86400"] line in the same indentation.
Example output
```
deployment.apps/<quay_database> edited
```

Execute the following command inside of your <quay_database>:

$ oc exec -it <quay_database> -- cat /var/lib/pgsql/data/userdata/postgresql/logs/* /path/to/desired_directory_on_host

Checking the connectivity between Project Quay and the database pod

To check the connectivity between your Project Quay instance and the database pod and troubleshoot connection issues, you can use oc exec for Operator deployments or podman exec for standalone deployments.

Procedure

Check the connectivity between Project Quay and the database pod.
1. If you are using the Project Quay Operator on OpenShift Container Platform, enter the following command:
  $ oc exec -it _quay_pod_name_ -- curl -v telnet://<database_pod_name>:5432
2. If you are using a standalone deployment of Project Quay, enter the following command:
  $ podman exec -it <quay_container_name >curl -v telnet://<database_container_name>:5432

Checking resource allocation

To check resource allocation for your Project Quay deployment and monitor disk, CPU, and memory usage, you can use oc exec or podman exec for disk usage and oc adm top pods or podman stats for other resources.

Procedure

Obtain a list of running containers.
Monitor disk usage of your Project Quay deployment.
1. If you are using the Project Quay Operator on OpenShift Container Platform, enter the following command:
  $ oc exec -it <quay_database_pod_name> -- df -ah
2. If you are using a standalone deployment of Project Quay, enter the following command:
  $ podman exec -it <quay_database_conatiner_name> df -ah
Monitor other resource usage.
1. Enter the following command to check resource allocation on a Project Quay Operator deployment:
  $ oc adm top pods
2. Enter the following command to check the status of a specific pod on a standalone deployment of Project Quay:
  $ podman pod stats <pod_name>
3. Enter the following command to check the status of a specific container on a standalone deployment of Project Quay:
  $ podman stats <container_name>
  The following information is returned:
  - CPU %. The percentage of CPU usage by the container since the last measurement. This value represents the container’s share of the available CPU resources.
  - MEM USAGE / LIMIT. The current memory usage of the container followed by its memory limit. The values are displayed in the format current_usage / memory_limit. For example, 300.4MiB / 7.795GiB indicates that the container is currently using 300.4 megabytes of memory out of a limit of 7.795 gigabytes.
  - MEM %. The percentage of memory usage by the container in relation to its memory limit.
  - NET I/O. The network I/O (input/output) statistics of the container. It displays the amount of data transmitted and received by the container over the network. The values are displayed in the format: transmitted_bytes / received_bytes.
  - BLOCK I/O. The block I/O (input/output) statistics of the container. It represents the amount of data read from and written to the block devices (for example, disks) used by the container. The values are displayed in the format read_bytes / written_bytes.

Troubleshooting Project Quay object storage

To troubleshoot Project Quay object storage and resolve issues with container image storage, you can use the procedures in this section. Object storage manages data as discrete units called objects, each with a unique identifier and metadata.

Unlike traditional file systems that organize data into hierarchical directories and files, object storage treats data as independent entities. Each object contains the data itself, along with metadata that describes the object and enables efficient retrieval.

Project Quay uses object storage as the underlying storage mechanism for storing and managing container images. It stores container images as individual objects. Each container image is treated as an object, with its own unique identifier and associated metadata.

Troubleshooting Project Quay object storage issues

To troubleshoot Project Quay object storage issues, you can check the QuayRegistry CR and config.yaml file, verify supported storage and network connectivity, enable debug mode, and test storage access outside Quay.

Procedure

Enter the following command to see what object storage is used:
```
$ oc get quayregistry quay-registry-name -o yaml
```
Ensure that the object storage you are using is officially supported by Project Quay by checking the tested integrations page.
Enable debug mode.
Check your object storage configuration in your config.yaml file. Ensure that the configuration is accurate and matches the settings provided by your object storage provider. You can check information like access credentials, endpoint URLs, bucket and container names, and other relevant configuration parameters.
Ensure that Project Quay has network connectivity to the object storage endpoint. Check the network configurations to ensure that no restrictions block communication between Project Quay and the object storage endpoint.

If FEATURE_STORAGE_PROXY is enabled in your config.yaml file, check to see if its download URL is accessible. This can be found in the Project Quay debug logs. For example:

$ curl -vvv "https://QUAY_HOSTNAME/_storage_proxy/dhaWZKRjlyO......Kuhc=/https/quay.hostname.com/quay-test/datastorage/registry/sha256/0e/0e1d17a1687fa270ba4f52a85c0f0e7958e13d3ded5123c3851a8031a9e55681?AWSAccessKeyId=xxxx&Signature=xxxxxx4%3D&Expires=1676066703"

Try access the object storage service outside of Project Quay to determine if the issue is specific to your deployment, or the underlying object storage. You can use command line tools like aws, gsutil, or s3cmd provided by the object storage provider to perform basic operations like listing buckets, containers, or uploading and downloading objects. This might help you isolate the problem.

Troubleshoot geo-replication and mirroring

Troubleshoot geo-replication and repository mirroring, including backend bucket checks and authentication.

Troubleshooting geo-replication for Project Quay

To troubleshoot geo-replication for Project Quay and resolve replication issues, you can use the procedures in the following sections. These procedures enable you to identify and fix problems with geo-replication deployments.

Checking data replication in backend buckets

To ensure that your Project Quay data is properly replicated in all backend buckets, you can use the aws CLI to list objects in the bucket. Run aws s3 ls with --recursive, --human-readable, and --summarize to verify replication.

Prerequisites

You have installed the aws CLI.

Procedure

Enter the following command to ensure that your data is replicated in all backend buckets:

$ aws --profile quay_prod_s3 --endpoint=http://10.0.x.x:port s3 ls ocp-quay --recursive --human-readable --summarize

Example output

Total Objects: 17996
Total Size: 514.4 GiB

Checking the status of your backend storage

To check the status of your Project Quay backend storage and verify access, you can use provider dashboards and CLIs for AWS, GCS, NooBaa, ODF, Ceph, Azure, or OpenStack Swift. Ensure all Project Quay instances have access to all S3 storage backends.

Amazon Web Service Storage (AWS). Check the AWS S3 service health status on the AWS Service Health Dashboard. Validate your access to S3 by listing objects in a known bucket using the aws CLI or SDKs.
Google Cloud Storage (GCS). Check the Google Cloud Status Dashboard for the status of the GCS service. Verify your access to GCS by listing objects in a known bucket using the Google Cloud SDK or GCS client libraries.
NooBaa. Check the NooBaa management console or administrative interface for any health or status indicators. Ensure that the NooBaa services and related components are running and accessible. Verify access to NooBaa by listing objects in a known bucket using the NooBaa CLI or SDK.
Red Hat OpenShift Data Foundation. Check the OpenShift Container Platform Console or management interface for the status of the Red Hat OpenShift Data Foundation components. Verify the availability of Red Hat OpenShift Data Foundation S3 interface and services. Ensure that the Red Hat OpenShift Data Foundation services are running and accessible. Validate access to Red Hat OpenShift Data Foundation S3 by listing objects in a known bucket using the appropriate S3-compatible SDK or CLI.
Ceph. Check the status of Ceph services, including Ceph monitors, OSDs, and RGWs. Validate that the Ceph cluster is healthy and operational. Verify access to Ceph object storage by listing objects in a known bucket using the appropriate Ceph object storage API or CLI.
Azure Blob Storage. Check the Azure Status Dashboard to see the health status of the Azure Blob Storage service. Validate your access to Azure Blob Storage by listing containers or objects using the Azure CLI or Azure SDKs.
OpenStack Swift. Check the OpenStack Status page to verify the status of the OpenStack Swift service. Ensure that the Swift services, like the proxy server, container servers, object servers, are running and accessible. Validate your access to Swift by listing containers or objects using the appropriate Swift CLI or SDK.

After checking the status of your backend storage, ensure that all Project Quay instances have access to all s3 storage backends.

Additional resources

Verifying authentication and permissions

To verify authentication and permissions for Project Quay repository mirroring and resolve mirroring issues, you can ensure credentials have the necessary access on source and destination instances.

On the Project Quay UI, check the following settings:

The access control settings. Ensure that the user or service account performing the mirroring operation has the required privileges.
The permissions of your robot account on the Project Quay registry.

Additional resources

Troubleshoot Clair scanning issues

Troubleshoot Clair scanning issues to restore vulnerability reporting in your registry.

Troubleshooting Clair issue

To troubleshoot Clair vulnerability scanning in Project Quay and resolve scan issues, you can verify image compatibility, allowlist Clair updaters when using a proxy, check the Clair config.yaml, and inspect image metadata.

Verifying image compatibility

If you are using Clair, ensure that Clair supports the images you are trying to scan. Clair has certain requirements and does not support all image formats or configurations.

Allowlisting Clair updaters

If you are using Clair behind a proxy configuration, you must allowlist the updaters in your proxy or firewall configuration.

Checking Clair configuration

Check your Clair config.yaml file to ensure that no misconfigurations or inconsistencies could lead to issues.

Inspect image metadata

In some cases, you might receive an Unsupported message. This might indicate that the scanner is unable to extract the necessary metadata from the image. Check if the image metadata is properly formatted and accessible.

Additional resources

Troubleshoot build failures

Troubleshoot Red Hat Quay build failures on Amazon EC2, OpenShift, and Kubernetes, including debug configuration flags.

Troubleshooting Builds

Builder instances in Project Quay are ephemeral and shut down automatically after builds complete or fail. To troubleshoot build issues, you must capture logs while the builds are running.

Debug configuration flag

The DEBUG configuration flag in Project Quay prevents builder instances from shutting down automatically after builds complete or fail. Set this flag to True to debug builder node issues.

  EXECUTORS:
    - EXECUTOR: ec2
      DEBUG: true
      ...
    - EXECUTOR: kubernetes
      DEBUG: true
      ...

When set to True, the debug feature prevents the build nodes from shutting down after the quay-builder service is done or fails. It also prevents the build manager from cleaning up the instances by terminating EC2 instances or deleting Kubernetes jobs. This allows debugging builder node issues.

Debugging should not be set in a production cycle. The lifetime service still exists; for example, the instance still shuts down after approximately two hours. When this happens, EC2 instances are terminated and Kubernetes jobs are completed.

Enabling debug also affects the ALLOWED_WORKER_COUNT because the unterminated instances and jobs still count toward the total number of running workers. As a result, the existing builder workers must be manually deleted if ALLOWED_WORKER_COUNT is reached to be able to schedule new builds.

Troubleshooting Amazon EC2

To troubleshoot Amazon EC2 builds in Project Quay, you can identify the builder instance in the EC2 console and access it via SSH. Check the quay-builder service logs to diagnose build issues.

Procedure

Start a Build in Project Quay.
In the EC2 console, identify the Build instance. Build instances are named Quay Ephemeral Builder and have the tag {<Build_UUID>: <uuid>}
Using the SSH key set by the EC2_KEY_NAME configuration field, log in to the Builder instance by running the following command:
```
$ ssh -i /path/to/ssh/key/in/ec2/or/config/id_rsa core@<instance_ip>
```
Obtain the quay-builder service logs by entering the following commands:
```
$ systemctl status quay-builder
```
```
$ journalctl -f -u quay-builder
```

Troubleshooting OpenShift Container Platform and Kubernetes Builds

To troubleshoot OpenShift Container Platform and Kubernetes builds in Project Quay, you can create a port forwarding tunnel to the builder pod and access it via SSH. Check the quay-builder service logs to diagnose build issues.

Procedure

Create a port forwarding tunnel between your local machine and a pod running with either an OpenShift Container Platform cluster or a Kubernetes cluster by entering the following command:
```
$ oc port-forward <builder_pod> 9999:2222
```
Establish an SSH connection to the remote host using a specified SSH key and port, for example:
```
$ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost
```
Obtain the quay-builder service logs by entering the following commands:
```
$ systemctl status quay-builder
```
```
$ journalctl -f -u quay-builder
```

Troubleshoot Operator-managed PostgreSQL TLS

Troubleshoot Operator-managed PostgreSQL TLS configurations to restore secure database connectivity.

Troubleshooting Operator-managed PostgreSQL TLS configurations

If TLS configuration is invalid, the Operator sets a RolloutBlocked condition on the QuayRegistry and emits Kubernetes Events with details.

Common causes include:

A referenced TLS Secret does not exist
The Secret is missing ca.crt, tls.crt, or tls.key
The certificate and private key do not match
The certificate has expired

On OpenShift Container Platform, if the Service CA serving certificate Secret has not yet been created, the Operator emits Events such as a PostgreSQL service CA error and retries until the Secret is available. A brief RolloutBlocked condition during that wait is expected.

If SHOW ssl; returns on but pg_stat_ssl shows no TLS client sessions, PostgreSQL is accepting encrypted connections, but Project Quay or Clair might still be connecting without SSL. Check the registry configuration for an sslmode value on the database URI or connection arguments, and review Operator Events for certificate or CA problems.

For external PostgreSQL databases, continue to configure TLS through DB_URI and related configuration fields in the config bundle. For more information, see Database configuration fields.