zppy weekly testing log #634

forsyth2 · 2024-10-18T15:55:17Z

forsyth2
Oct 18, 2024
Maintainer

Starting this page to log the process/results for the weekly runs of zppy's integration testing: bundles, comprehensive_v2, comprehensive_v3

forsyth2 · 2024-10-18T16:28:57Z

forsyth2
Oct 18, 2024
Maintainer Author

Friday 2024-10-18

Note that before we get to testing a release candidate for Unified, we'll need to get v3 test data on to Perlmutter and Compy too.

On Chrysalis:

# Set up branch
cd ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241018 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/15: make --closedcontourcmd resolution specific (#824)
mamba clean --all
# Press `y` twice
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241018
conda activate e3sm_diags_20241018
pip install .

# Set up testing files
cd ../zppy
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241018"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241018",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241018 
conda activate zppy_dev_weekly_20241018 
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# Once that finishes:
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241018/v2.LR.historical_0201/post/scripts
grep -v "OK" *status

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241018/v3.LR.historical_0051/post/scripts
grep -v "OK" *status

# Run integration tests
python -u -m unittest tests/integration/test_*.py

This gets us these results:

======================================================================
FAIL: test_bash_generation (tests.integration.test_bash_generation.TestBashGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_bash_generation.py", line 14, in test_bash_generation
    self.assertEqual(
AssertionError: 256 != 0

----------------------------------------------------------------------
Ran 14 tests in 1291.690s

FAILED (failures=1)

python -u -m unittest tests/integration/test_bash_generation.py

We can see the diff is from the formatting change introduced in #627, so this is expected. Let's update the expected results.

cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20241018
# Currently, copying over entire expected results directory to keep a snapshot history 
# But we could just copy over subdirs that changed in a given week.

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh

That gives:

----------------------------------------------------------------------
Ran 1 test in 0.835s

OK

✅ This week's testing of main is successful.

0 replies

forsyth2 · 2024-10-25T19:12:52Z

forsyth2
Oct 25, 2024
Maintainer Author

Friday 2024-10-25

Commits since 2024-10-18:
zppy:

None

e3sm_diags:

Fixing African easterly wave density plots in TC analysis (Fixing African easterly wave density plots in TC analysis e3sm_diags#851)

# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241025 upstream/main
git log
# Last commit, from 10/15: Improve carryover dependency handling (#623)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/23: Fixing African easterly wave density plots in TC analysis (#851)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241025
conda activate e3sm_diags_20241025
pip install .

# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241025"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241025 # This command took a while
conda activate zppy_dev_weekly_20241025
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 22 tests pass

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241025
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241025
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.o614397 
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/lnd/180x360_aave/clim/2yr/v3.LR.historical_0051_*_1987??_1988??_climo.nc': No such file or directory

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.status:ERROR (1)
cat e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1982-1983_vs_1980-1981.o614424 
# cp: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241025/v2.LR.historical_0201/post/lnd/180x360_aave/clim/2yr/v2.LR.historical_0201_*_1982??_1983??_climo.nc': No such file or directory

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241025/v3.LR.historical_0051/post/scripts
grep -v "OK" *status

# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py

That gives:

======================================================================
ERROR: test_comprehensive_v2_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 36, in test_comprehensive_v2_images
    test_images(self, "comprehensive_v2", V2_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 99, in check_mismatched_images
    compare_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 65, in compare_images
    shutil.copy(
  File "/home/ac.forsyth2/miniconda3/envs/zppy_dev_weekly_20241025/lib/python3.9/shutil.py", line 427, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/ac.forsyth2/miniconda3/envs/zppy_dev_weekly_20241025/lib/python3.9/shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20241025/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/tc_analysis/aew-density-map.png_actual.png'

======================================================================
FAIL: test_comprehensive_v3_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 39, in test_comprehensive_v3_images
    test_images(self, "comprehensive_v3", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 121, in check_mismatched_images
    test.assertEqual(missing_images, [])
AssertionError: Lists differ: ['e3sm_diags/lnd_monthly_mvm_lnd/model_vs_[43955 chars]png'] != []

First list contains 336 additional elements.
First extra element 0:
'e3sm_diags/lnd_monthly_mvm_lnd/model_vs_model_1987-1988/viewer/viewer/e3sm_logo.png'

Diff is 45015 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 14 tests in 983.368s

FAILED (failures=1, errors=1)

The tc_analysis error is likely because of the new commit in e3sm_diags, so that will likely just be a matter of updating the expected results.

The lnd_monthly_mvm_lnd error is to be expected, considering the cannot stat error detected earlier.

Let's look at the image check failures:

v3 is empty.
v2 is empty.

Let's try the directory paths:

$ ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_weekly_20241025/v3.LR.historical_0051/image_c
heck_failures_comprehensive_v3
# Empty

$ ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_zppy_weekly_20241025/v2.LR.historical_0201/image_c
heck_failures_comprehensive_v2
# Empty

Clearly, image check failures aren't being routed to the right place.

1 reply

forsyth2 Oct 25, 2024
Maintainer Author

Bug report is at #635.

forsyth2 · 2024-11-01T21:21:37Z

forsyth2
Nov 1, 2024
Maintainer Author

Friday 2024-11-01

Commits since 2024-10-25:
zppy:

Fix dir issues in comparing images (Fix dir issues in comparing images #638)
Updates to diags handling (Update to diags handling #633)

e3sm_diags:

Convert H2OLNZ units to ppm by volume (Convert H2OLNZ units to ppm by volume e3sm_diags#874)
add_TCO_60S60N to default lat-lon (https://github.com/E3SM-Project/e3sm_diags/pull/#879)

# Set up branch
cd ~/ez/zppy
git status
# Check no file changes will persist when we switch branches.
git fetch upstream main
git checkout -b test_zppy_weekly_20241101 upstream/main
git log
# Last commit, from 10/30: Updates to diags handling (#633)

# Set up diags to use
cd ../e3sm_diags
git checkout main
git status
# Check no file changes will persist when we switch branches.
git fetch upstream
git reset --hard upstream/main
git log
# Last commit, from 10/29: add_TCO_60S60N to default lat-lon (#879)
mamba clean --all --y
mamba env create -f conda-env/dev.yml -n e3sm_diags_20241101
conda activate e3sm_diags_20241101
pip install .

# Set up testing files
cd ../zppy
# NOTE: CURRENTLY MANUAL STEP --could potentially pass in arguments to utils.py?
# Edit tests/integration/utils.py:
# UNIQUE_ID = "test_zppy_weekly_20241101"
# For get_chyrsalis_expansions:         "diags_environment_commands": "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101",
# Keep this as is:     generate_cfgs(unified_testing=False, dry_run=False)

# Set up zppy environment
mamba clean --all --y
mamba env create -f conda/dev.yml -n zppy_dev_weekly_20241101 # This command took a while
conda activate zppy_dev_weekly_20241101
pip install .

# Run unit tests
python -u -m unittest tests/test_*.py
# All 23 tests pass (increased from 22)

# Run zppy to produce actual results to compare in the integration tests
python tests/integration/utils.py
# That prints:
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=test_zppy_weekly_20241101
# unified_testing=False
# diags_environment_commands=source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_20241101
# e3sm_to_cmip_environment_commands=
# environment_commands=
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that those 3 zppy runs have finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once those all finish: (took about 1h15m)
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg

# NOTE: CURRENTLY MANUAL STEP -- check that that zppy run has finished. Try bash `wait command? Just keep running `squeue` every hour until there are no jobs left?
# Once that finishes: (took about 10m)
# Check output
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_weekly_20241101/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_weekly_20241101/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# No errors

# Run integration tests
cd ~/ez/zppy
python -u -m unittest tests/integration/test_*.py

That gives:

======================================================================
FAIL: test_bash_generation (tests.integration.test_bash_generation.TestBashGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_bash_generation.py", line 14, in test_bash_generation
    self.assertEqual(
AssertionError: 256 != 0

======================================================================
FAIL: test_comprehensive_v2_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 36, in test_comprehensive_v2_images
    test_images(self, "comprehensive_v2", V2_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['e3sm_diags/atm_monthly_180x360_aave_mvm/[2858 chars]png'] != []

First list contains 25 additional elements.
First extra element 0:
'e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1980-1981/tc_analysis/aew-density-map.png'

Diff is 2985 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_comprehensive_v3_images (tests.integration.test_weekly.TestWeekly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 39, in test_comprehensive_v3_images
    test_images(self, "comprehensive_v3", V3_CASE_NAME)
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/test_weekly.py", line 27, in test_images
    check_mismatched_images(
  File "/gpfs/fs1/home/ac.forsyth2/ez/zppy/tests/integration/utils.py", line 132, in check_mismatched_images
    test.assertEqual(mismatched_images, [])
AssertionError: Lists differ: ['e3sm_diags/atm_monthly_180x360_aave_mvm/[2019 chars]png'] != []

First list contains 16 additional elements.
First extra element 0:
'e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1987-1988/zonal_mean_xy/model_vs_model/v3.LR.historical_0051-Z3-500-SON-global.png'

Diff is 2119 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 14 tests in 1691.117s

FAILED (failures=3)

python -u -m unittest tests/integration/test_bash_generation.py

The diff is from the template file change in https://github.com/E3SM-Project/zppy/pull/633/files#diff-fbd1002ca5f9034c321e896884792a69fc7e4a9fc14e826a3954dc81a3d6d76d, so this is expected. Let's update the expected results.

cd /lcrc/group/e3sm/public_html
cp -r zppy_test_resources zppy_test_resources_previous/expected_results_until_20241101
# Currently, copying over entire expected results directory to keep a snapshot history 
# But we could just copy over subdirs that changed in a given week.

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh

That test passes now.

As for the image check failures (attempted to format this hierarchically, but only semi-successfully):

v2 diags
1.1 model_vs_obs
1.1.1. tc_analysis. These changes are from Fixing African easterly wave density plots in TC analysis e3sm_diags#851 (that was merged last week, but expected results weren't updated because of [Bug]: Errors in 2024-10-25 test of main #635).
1.1.2. zonal_mean_2d_stratosphere. The ERA5 and MERRA2 diffs appear to be from Convert H2OLNZ units to ppm by volume e3sm_diags#874 and thus expected.
1.1.2.1. ERA5
1.1.2.2. MERRA2
1.2. model_vs_model
1.2.1. tc_analysis. Again, from Fixing African easterly wave density plots in TC analysis e3sm_diags#851.
1.2.2. zonal_mean_2d_stratosphere. Here, the issue is with the Q plots, but still from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
v3 diags
2.1. model_vs_obs zonal_mean_2d_stratosphere. Again, from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
2.1.1. ERA5
2.1.2. MERRA2
2.2. model_vs_model
2.2.1. zonal_mean_2d_stratosphere. Again, from Convert H2OLNZ units to ppm by volume e3sm_diags#874.
2.2.2. zonal_mean_xy. This has one diff, which actually appears blank. It doesn't look like zonal_mean_xy has been modified recently. I suspect this is just a very minor matplotlib pixel shift that has appeared randomly -- not a concern.

So, all image check failures are accounted for. Let's update the expected results.

# We already copied over the previous expected results

cd /home/ac.forsyth2/ez/zppy
chmod u+x tests/integration/generated/update_weekly_expected_files_chrysalis.sh
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh

After about 20 minutes, that gives:

.
----------------------------------------------------------------------
Ran 5 tests in 1298.721s

OK

All expected results have been updated.

✅ This week's testing of main is successful.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zppy weekly testing log #634

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

zppy weekly testing log #634

forsyth2 Oct 18, 2024 Maintainer

Replies: 3 comments · 1 reply

forsyth2 Oct 18, 2024 Maintainer Author

Friday 2024-10-18

forsyth2 Oct 25, 2024 Maintainer Author

Friday 2024-10-25

forsyth2 Oct 25, 2024 Maintainer Author

forsyth2 Nov 1, 2024 Maintainer Author

Friday 2024-11-01

forsyth2
Oct 18, 2024
Maintainer

Replies: 3 comments 1 reply

forsyth2
Oct 18, 2024
Maintainer Author

forsyth2
Oct 25, 2024
Maintainer Author

forsyth2 Oct 25, 2024
Maintainer Author

forsyth2
Nov 1, 2024
Maintainer Author