Testing Procedures

Contents

• 1   Purpose of this page
• 2   Template
  • 2.1   Repository location
  • 2.2   How to checkout the code for testing
  • 2.3   Test commands to run
  • 2.4   Baseline location(s)
  • 2.5   How to determine expected failures
  • 2.6   Does full testing need to be run on each set of changes?
  • 2.7   Does a branch need to be up-to-date with the latest main branch before final testing?
  • 2.8   Rough estimate of test turnaround time and core-hour cost
  • 2.9   Any other important notes about the testing process
  • 2.10   Where to go for more information

1   Purpose of this page

This page gives a brief overview of the testing procedures for each component / repository that makes up CESM. The audience is CESM software engineers who are familiar with CESM / CIME system testing in general, but who need to run testing on a component that they do not regularly work with.

Many components have different testing procedures that are run depending on what has changed. This page is not meant to be exhaustive, but rather is meant to capture the most common testing procedures – e.g., procedures that should cover about 90% of cases.

2   Template

[This section provides a template that can be copied to provide a starting point for the documentation of a given component's testing procedure. Replace "Template" above with the component's name. Any text in brackets – like this overview text – should be removed from the final version.]

2.1   Repository location

[This should give the GitHub URL of the repository. This is assumed to be the location into which a Pull Request should be opened.]

2.2   How to checkout the code for testing

[Does this component support a standalone checkout for testing, or does it need to be checked out in the context of CESM or some other "umbrella" repository? If the latter, describe how to determine a reasonable version of the umbrella repository to use for testing – and in particular, how to determine what versions will have baselines available for system tests of the component.]

2.3   Test commands to run

[Give details of the exact test commands to run to achieve full system testing, and on what machine(s). As noted above, this does not need to exhaustively cover all possible testing scenarios. Rather, it is meant to cover the typical cases – say, the common 90% of scenarios. The testing commands should be detailed enough that someone can essentially copy and paste the commands into a terminal (replacing some placeholder text, e.g., for the baseline tag name to be used for comparison.]

2.4   Baseline location(s)

[For each test machine, give the location on disk in which baselines can be found for the above testing.]

2.5   How to determine expected failures

[Describe how expected failures can be determined for any given tag.]

2.6   Does full testing need to be run on each set of changes?

[Does the full testing described above need to be run on each set of changes? Or is it common to combine multiple changes / PRs together and have a component SE run full testing on the batch of changes? If the latter, is there some minimum expected / recommended testing on the individual changes?]

2.7   Does a branch need to be up-to-date with the latest main branch before final testing?

[State whether a feature branch needs to merge in the latest version of the main branch before running final testing. If so, there will typically be some process for tag ordering, so briefly describe that process.]

2.8   Rough estimate of test turnaround time and core-hour cost

[Give a very rough estimate of test turnaround time and core-hour cost, to help answer the question: if I run this test suite a number of times, am I going to burn through my allocation?]

2.9   Any other important notes about the testing process

[Add anything else that is important but not covered above.]

2.10   Where to go for more information

[Provide URLs for additional documentation on this component's testing procedures.]