Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sfs as valid system #3243

Open
wants to merge 16 commits into
base: develop
Choose a base branch
from
Open

Add sfs as valid system #3243

wants to merge 16 commits into from

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Jan 21, 2025 •
edited
Loading

Description

Adds sfs as a valid option for NET.

To start, the GEFS system is generally just copied wholesale for SFS. This includes the extract_vars job.

Other than base and resources, config files link to the GEFS versions, just as GEFS config files point to the GFS versions except where they have needed to be changed.

The temporary SFS_POST option has been removed.

The existing SFS test is copied and slightly modified for a PR-level CI test.

Resolves #2271

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? YES
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • gfs cycled test on Hercules
  • gefs test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@WalterKolczynski-NOAA
Copy link
Contributor Author

@NeilBarton-NOAA would you please supply ICs for the SFS test so we can set it up to run on all systems

@WalterKolczynski-NOAA
Copy link
Contributor Author

Still need to add SFS to the RTD documentation.

@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as draft January 21, 2025 19:19
@aerorahul aerorahul requested a review from pjpegion January 21, 2025 19:26
@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as ready for review January 23, 2025 03:22
@WalterKolczynski-NOAA
Copy link
Contributor Author

ICs are in place and documentation has been updated. Ready for review.

EricSinsky-NOAA
EricSinsky-NOAA requested changes Jan 28, 2025
View reviewed changes
Copy link
Contributor

@EricSinsky-NOAA EricSinsky-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are a couple of suggestions.

workflow/applications/sfs.py Outdated Show resolved Hide resolved
workflow/applications/sfs.py Show resolved Hide resolved
workflow/rocoto/sfs_tasks.py Outdated Show resolved Hide resolved
workflow/rocoto/sfs_tasks.py Outdated Show resolved Hide resolved
workflow/rocoto/sfs_tasks.py Outdated Show resolved Hide resolved
parm/config/sfs/config.base Outdated Show resolved Hide resolved
parm/config/sfs/config.base Show resolved Hide resolved
@WalterKolczynski-NOAA
Add sfs as valid system
f5ec7ec 
Adds sfs as a valid option for NET.

To start, the GEFS system is generally just copied wholesale for SFS.
This includes the extract_vars job.

Other than base and resources, config files link to the GEFS versions,
just as GEFS config files point to the GFS versions except where they
have needed to be changed.

The temporarily SFS_POST option has been removed.

The sfs test case is moved from a separate sfs directory to the pr
directory, but ICs are still needed before it is functional.

Resolves NOAA-EMC#2271
@WalterKolczynski-NOAA
Update docs for new sfs option
103a5ef
@WalterKolczynski-NOAA
Add missing sfs option to CICE postdet
87ac6a3
@WalterKolczynski-NOAA
Restore old sfs case and make new one for PRs
5a5cd80 
The SFS case is moved back to its original position in an sfs directory,
then a copy is made in the PR directory using a shorter time period and
a couple other changes.
@WalterKolczynski-NOAA
Add ensemble grib header info for SFS
5b39079
@WalterKolczynski-NOAA
Turn off SFS archival, add SFS fh computation
44c264e 
Adds SFS to the list of supported runs for `Tasks._get_forecast_hours()`

Turns off the archive job for SFS for the time being until an appropriate
yaml is created.
EricSinsky-NOAA
EricSinsky-NOAA previously approved these changes Jan 30, 2025
View reviewed changes
Copy link
Contributor

@EricSinsky-NOAA EricSinsky-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you for adding sfs as a valid option in gw.

@WenMeng-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA Would you like to include sfs-specified atmos. product generation (pgrb2 files) in this PR? If so, I can provide the product list.

@WalterKolczynski-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA Would you like to include sfs-specified atmos. product generation (pgrb2 files) in this PR? If so, I can provide the product list.

@WenMeng-NOAA Is it just linking in new files from UPP, or is there additional work?

@WenMeng-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA Would you like to include sfs-specified atmos. product generation (pgrb2 files) in this PR? If so, I can provide the product list.

@WenMeng-NOAA Is it just linking in new files from UPP, or is there additional work?

@WalterKolczynski-NOAA The product list is sfs specified, similar to gfs product list. I created this list based on the UPP output for sfs and would suggest the product generation process only generates one dataset "pgrb2" in 1p00 deg. instead of two datasets "pgrb2" and "pgrb2b". There might be additional work for you to update relevant scripts for sfs.

@WalterKolczynski-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA Would you like to include sfs-specified atmos. product generation (pgrb2 files) in this PR? If so, I can provide the product list.

@WenMeng-NOAA Is it just linking in new files from UPP, or is there additional work?

@WalterKolczynski-NOAA The product list is sfs specified, similar to gfs product list. I created this list based on the UPP output for sfs and would suggest the product generation process only generates one dataset "pgrb2" in 1p00 deg. instead of two datasets "pgrb2" and "pgrb2b". There might be additional work for you to update relevant scripts for sfs.

I'm going to leave that for a future PR then. This is just to get the SFS framework in.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 6, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Feb 6, 2025
@emcbot
Copy link

emcbot commented Feb 6, 2025

Experiment C48_S2SW FAILED on Hercules in Build# 6 with error logs:

/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48_S2SW_d58fb75b/logs/2021032312/gfs_wavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Feb 6, 2025
@emcbot
Copy link

emcbot commented Feb 6, 2025

Experiment C48_S2SW FAILED on Hercules in Build# 6 in
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/EXPDIR/C48_S2SW_d58fb75b

@emcbot
Copy link

emcbot commented Feb 7, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 6 with error logs:

/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d58fb75b/logs/2021032500/gfs_cleanup.log
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d58fb75b/logs/2021032500/gfs_wavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Feb 7, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 6 in
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d58fb75b

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 7, 2025
@emcbot
Copy link

emcbot commented Feb 7, 2025

CI Failed on Hercules in Build# 6
Built and ran in directory /work2/noaa/global/CI/HERCULES/3243


Experiment C48_ATM_d58fb75b Completed 1 Cycles: *SUCCESS* at Thu Feb  6 17:24:02 CST 2025
Experiment C96mx100_S2S_d58fb75b Completed 1 Cycles: *SUCCESS* at Thu Feb  6 17:24:12 CST 2025
Experiment C48_S2SW_d58fb75b Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Feb  6 17:36:27 CST 2025
Experiment C48_S2SW_d58fb75b Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48_S2SW_d58fb75b/logs/2021032312/gfs_wavepostpnt.log
Experiment C48mx500_hybAOWCDA_d58fb75b Completed 2 Cycles: *SUCCESS* at Thu Feb  6 18:06:35 CST 2025
Experiment C48mx500_3DVarAOWCDA_d58fb75b Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Feb  6 18:24:54 CST 2025
Experiment C48mx500_3DVarAOWCDA_d58fb75b Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d58fb75b/logs/2021032500/gfs_cleanup.log
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d58fb75b/logs/2021032500/gfs_wavepostpnt.log
Experiment C96_atm3DVar_d58fb75b Completed 3 Cycles: *SUCCESS* at Thu Feb  6 19:25:19 CST 2025
Experiment C96C48_hybatmDA_d58fb75b Completed 3 Cycles: *SUCCESS* at Thu Feb  6 19:25:19 CST 2025
Experiment C48_S2SWA_gefs_d58fb75b Completed 1 Cycles: *SUCCESS* at Thu Feb  6 20:20:54 CST 2025

@WalterKolczynski-NOAA
Copy link
Contributor Author

Will look closer tomorrow, but I think cleanup may be triggering before wave post point is done.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 10, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Feb 10, 2025
@emcbot
Copy link

emcbot commented Feb 10, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 7 with error logs:

/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_9f250971/logs/2021032500/gdas_verfrad.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Feb 10, 2025
@emcbot
Copy link

emcbot commented Feb 10, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 7 in
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_9f250971

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 10, 2025
@emcbot
Copy link

emcbot commented Feb 10, 2025

CI Failed on Hercules in Build# 7
Built and ran in directory /work2/noaa/global/CI/HERCULES/3243


Experiment C96mx100_S2S_9f250971 Completed 1 Cycles: *SUCCESS* at Mon Feb 10 14:37:22 CST 2025
Experiment C48_ATM_9f250971 Completed 1 Cycles: *SUCCESS* at Mon Feb 10 14:43:21 CST 2025
Experiment C48mx500_3DVarAOWCDA_9f250971 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Feb 10 15:13:58 CST 2025
Experiment C48mx500_3DVarAOWCDA_9f250971 Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3243/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_9f250971/logs/2021032500/gdas_verfrad.log
Experiment C48mx500_hybAOWCDA_9f250971 Completed 2 Cycles: *SUCCESS* at Mon Feb 10 15:19:46 CST 2025
Experiment C96C48_hybatmDA_9f250971 Completed 3 Cycles: *SUCCESS* at Mon Feb 10 15:50:11 CST 2025
Experiment C96_atm3DVar_9f250971 Completed 3 Cycles: *SUCCESS* at Mon Feb 10 16:02:12 CST 2025
Experiment C48_S2SW_9f250971 Completed 1 Cycles: *SUCCESS* at Mon Feb 10 16:26:16 CST 2025
Experiment C48_S2SWA_gefs_9f250971 Completed 1 Cycles: *SUCCESS* at Mon Feb 10 16:57:08 CST 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA left review comments

@XiaqiongZhou-NOAA XiaqiongZhou-NOAA XiaqiongZhou-NOAA approved these changes

@aerorahul aerorahul aerorahul left review comments

@EricSinsky-NOAA EricSinsky-NOAA EricSinsky-NOAA left review comments

@NeilBarton-NOAA NeilBarton-NOAA NeilBarton-NOAA left review comments

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA is a code owner

@pjpegion pjpegion Awaiting requested review from pjpegion

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add SFS configuration
9 participants
@WalterKolczynski-NOAA @WenMeng-NOAA @emcbot @TerrenceMcGuinness-NOAA @aerorahul @XiaqiongZhou-NOAA @EricSinsky-NOAA @DavidHuber-NOAA @NeilBarton-NOAA