Skip to content

Robottelo Pull Request Testing (PRT) Process

Jump to bottom
vsedmik edited this page Sep 21, 2023 · 24 revisions

Robottelo Pull Request Testing (PRT) Process Wiki

Welcome to the official wiki for the Robottelo project's Pull Request Testing (PRT) process. This wiki provides a comprehensive guide on how to use the PRT process to effectively test pull requests within the Robottelo project.

Table of Contents

Introduction to PRT in Robottelo

The Pull Request Testing (PRT) process in the Robottelo project aims to ensure the quality and stability of code changes introduced through pull requests. By following this process, contributors can collaborate effectively while maintaining a high standard of code.

PRT Process Steps

Step 1: Pull Request Submission

A contributor creates a pull request from their forked repository to the main Robottelo repository.

Step 2: PRT Comment Initiation

The PRT process is initiated by the PR author, who writes a PRT comment in the PR. This comment should include:

  • Trigger Statement: The comment should contain a trigger statement such as "trigger": "test-robottelo". This statement is essential for the Jenkins job to identify the request for automated testing.

  • Test Details: The PR author specifies the details of the tests to be run for the project. This can include information about which tests should be executed and any specific test configurations.

Step 3: Automated Testing

Once the PRT comment is added with the trigger statement and test details, the automated testing process is initiated as follows:

  • Test Trigger: The presence of the trigger statement in the comment automatically triggers the Jenkins job for automated testing as part of the pull request's CI/CD pipeline.

  • Test Execution: Jenkins executes the specified tests based on the provided details. This can include running specific test suites, using pytest options, or following other test configurations as defined in the comment.

Step 4: Review and Reporting

After the automated tests have been executed, the pull request proceeds to manual review.

Once the review is complete and all tests have passed, the results are reported using the following methods:

  • GitHub Status API: The test results are uploaded to GitHub's status API, providing a clear indication on the PR page that all tests have passed successfully.

  • Jenkins Build Number: The Jenkins build number associated with the PR is also displayed, allowing reviewers to cross-reference the build details if needed. This provides more insights into the testing process and results, ensuring a comprehensive review.

Usage Examples

Example 1: Basic PRT Comment

To initiate automated testing for this pull request, add the following comment:

"trigger": "test-robottelo"

This will trigger the PRT process, which will run the default test suite (Sanity Test Suite) for this Robottelo.

Example 2: Using pytest Options

If you want to run specific tests or use pytest options, you can provide the details in the PRT comment. For example:

"trigger": "test-robottelo"
"pytest": "-k 'test_positive_create_with_auth_puppet_repo' tests/foreman/"

Example 3: Running Tests for Airgun or Nailgun PRs

For pull requests related to Airgun or Nailgun, you can specify additional details in the PRT comment. Here's an example:

"trigger": "test-robottelo"
"pytest": "-k 'test_positive_create_with_auth_puppet_repo' tests/foreman/"
"airgun": "458"
"nailgun": "227"

Example 4: Running Tests for Provisioning Component

To initiate automated testing for this pull request, including provisioning tests, add the following comment:

"trigger": "test-robottelo"
"pytest": "tests/foreman/api/test_provisioning.py -k test_rhel_ipxe_provisioning"
"provisioning": true

This comment will trigger the PRT process, which will run the specified provisioning test using the provided pytest options.

Example 5: Running Tests for custom inventory and pod size

To initiate automated testing for this pull request with custom inventory and pod size, add the following comment:

"trigger": "test-robottelo"
"pytest": "-k 'test_positive_create_with_auth_puppet_repo' tests/foreman/"
"inventory": "<invntory name>"
"pod_resources_size": "large"

In this example:

  • `"inventory": indicates that an ansible inventory name from satlab
  • "pod_resources_size": "large" sets the pod resource size to 'large' for this testing session.

This comment will trigger the PRT process with the specified configurations, allowing you to customize the test environment based on your requirements.

Example 6: Running Tests with exporting the env variables

To initiate automated testing for this pull request on a specific environment and set environment variables, add the following comment:

trigger: test-robottelo
pytest: tests/foreman/ui/test_sync.py -k 'test_positive_sync_custom_repo'
env:
  ROBOTTELO_ui__record_video: true

In this example:

  • "env" is used to define environment variables, and "ROBOTTELO_ui__record_video": true sets the ROBOTTELO_ui__record_video environment variable to true.

This comment will trigger the PRT process with the specified configurations, allowing you to run tests on a specific environment with the defined environment variables.

Warning: Do not pass sensitive information or secrets as environment variables in PRT comments. Environment variables in PRT comments are visible to anyone with access to the pull request, and exposing sensitive information in this way can lead to security risks. Ensure that any environment variables used in testing do not contain confidential data or secrets.

Example 7: Running Upgrade All Tier Tests with PRT

To initiate automated testing for upgrade all tiers with specific configurations, add the following comment:

trigger: test-robottelo
pytest: tests/foreman/api/test_product.py -k test_positive_create_with_name
env:
    ROBOTTELO_server__deploy_workflow: 'deploy-satellite-upgrade'
    ROBOTTELO_server__deploy_arguments__deploy_rhel_version: '8'
    ROBOTTELO_server__deploy_arguments__deploy_sat_version: '6.14'
    ROBOTTELO_server__deploy_arguments__deploy_snap_version: 'ystream'
    BROKER_AnsibleTower__inventory: 'osp-rhos01-satellite-upgrade'

In this example:

  • "env" is used to define environment variables for the upgrade process, including the workflow, RHEL version, Satellite version, Snap version, and Ansible Tower inventory.

This comment will trigger the PRT process with the specified configurations, allowing you to perform an upgrade on all tiers with the defined environment variables.

FAQ

Q: What is the purpose of the Pull Request Testing (PRT) process in Robottelo?

A: The PRT process in Robottelo aims to ensure the quality and stability of code changes introduced through pull requests. It provides an automated and systematic approach to testing, making it easier for contributors to collaborate effectively while maintaining code quality.

Q: How do I initiate PRT for my pull request?

A: To initiate PRT for your pull request, add a PRT comment with the necessary configurations. Include a trigger statement like "trigger": "test-robottelo" and any additional parameters such as the specific tests to run or environment variables.

Q: Can I specify which tests to run in my pull request?

A: Yes, you can specify which tests to run by including test details in your PRT comment. For example, you can use "pytest": "tests/foreman/ui/test_sync.py -k 'test_positive_sync_custom_repo'" to specify the test file and test case.

Q: Is it possible to adjust the test environment for my PRT?

A: Yes, you can customize the test environment by using environment variables in your PRT comment. However, ensure that you do not pass sensitive information or secrets as environment variables to maintain security.

Q: What should I do if my pull request fails automated tests in PRT?

A: If your pull request fails automated tests, you should review the test results to identify the issues. Make the necessary code changes to address the problems, and once you believe the issues are resolved, you can re-trigger PRT for your pull request.

Q: How are PRT results reported and tracked?

A: PRT results are reported through the GitHub Status API and the Jenkins Build Number associated with the pull request. You can check the GitHub PR page for a clear indication of test results and access build details for further insights.

Q: Can I run PRT on specific environments or configurations?

A: Yes, you can use environment variables in your PRT comment to run tests on specific environments or with customized configurations. For example, you can set the RHEL version, Satellite version, or other parameters as needed.

Q: What should I do if I encounter issues or have questions about the PRT process?

A: If you encounter any issues or have questions related to the PRT process, you can reach out ##team-satellite-automation slack channel

Q: How much delay does the PRT process typically have before starting tests?

A: The PRT process includes a 5-minute delay before it starts picking up your pull request. This delay serves a couple of important purposes:

Resource Allocation: The delay allows the system to allocate the necessary resources, such as virtual machines or containers, for testing. This ensures that resources are available when testing begins.

Queue Management: The delay helps manage the testing queue efficiently. By introducing a brief delay, the system can avoid processing multiple pull requests simultaneously, which could lead to resource contention and slow down the testing process.

This intentional delay is a part of the normal PRT workflow and is designed to ensure that your pull request testing runs smoothly and without interruptions.

Feel free to modify these FAQ entries to fit your project's specific context and add any additional questions that you think would be relevant to your users.

Robottelo Wiki

Additional Resources

Clone this wiki locally