Question on model vs model e3sm_diags task zppy configuration #607

chengzhuzhang · 2024-06-19T17:17:35Z

chengzhuzhang
Jun 19, 2024
Maintainer

Question criteria

I added a descriptive title here.
I searched the zppy GitHub Discussions to find a similar question and didn't find it.
I searched the zppy documentation.

What is the deadline?

NA

Describe your question

@BunnyVon and I are working on a zppy configuration for compare a BGC run with a v2.amip run. The configuration file can be viewed here: /home/ac.zhang40/test_zppy/test_from_sha_618.cfg

We had this problem that the diurnal_cycle set from e3sm_diags model-vs-model run is not working. The error was '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_diurnal_8xdaily/30yr/v2.LR.amip_0101.eam.h3_*_1985??_2014??_[climo.nc](http://climo.nc/)': No such file or directory
However the diurnal files are saved under a path: /lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave/30yr, also the prefix for the file should be v2.LR.amip_0101.eam.h4, instead of v2.LR.amip_0101.eam.h3 as zppy is expecting. @forsyth2 not sure if there is a way to point to the proper files in this case?

Are there are any possible answers you came across?

No response

What machine were you running on?

Chrysalis

Environment

e3sm_unified 1.10.

What command did you run?

zppy -c /home/ac.zhang40/test_zppy/test_from_sha_618.cfg
(The script is currently working with `diurnal_cycle` removed from sets.)

Copy your cfg file

[default]
case = v2.LR.BGC-LNDATM.CONTRL
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
input = "/lcrc/group/e3sm2/ac.sfeng1/E3SM_Simulations/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
output = "/lcrc/group/e3sm2/ac.zhang40/tests/E3SM_Simulations/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
partition = "debug"
# qos = "regular"
qos = "debug"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/tests/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
walltime = "4:00:00"

[climo]
active = True
years = "1985:2014:30",

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h3"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

  [[ land_monthly_climo ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = archive/lnd/hist
  vars = ""

[ts]
active = True
years = "1985:2014:10",


  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS,CLDICE,CLDLIQ,T,AODDUST"
# Needed for mixed-phase partition
#  vars = "LANDFRAC,CLDICE,CLDLIQ,T" 
  ts_fmt = "cmip"

  [[ atm_daily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h1"
  frequency = "daily"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # Needed for Wheeler Kiladis
  vars = "FLUT,PRECT,U850"

  [[ atm_monthly_glb ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"

  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  frequency = "monthly"
  #mapping_file = ""
  mapping_file = map_r05_to_cmip6_180x360_aave.20231110.nc
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  extra_vars = "landfrac"
  ts_fmt = "cmip"
#
  [[ rof_monthly ]]
  input_subdir = "archive/rof/hist"
  input_files = "mosart.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
  extra_vars = 'areatotal2'

  [[ lnd_monthly_glb ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "glb"
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"

[e3sm_diags]
active = True
multiprocessing = True
partition = "debug"
nodes = 8
years = "1985:2014:30",
ts_num_years = 10
ref_start_yr = 1985
ref_final_yr = 2014

   [[atm_monthly_180x360_aave_mvm]]
   years = "1985-2014", 
   ref_years = "1985-2014",
   ref_start_yr = 1985
   ref_final_yr = 2014
   ts_num_years = 10   
   ts_num_years_ref = 5
   ts_subsection = "atm_monthly_180x360_aave"
   input_files = "eam.h3"
   short_name = 'BGCv2LNDATM'
   grid = '180x360_aave'
   ref_name = ' v2.LR.amip_0101'
   short_ref_name = 'v2.LR.amip'
  #  tag = 'v3alpha04i-COARE_vs_CTL'
   tag = "model_vs_model"
   run_type = "model_vs_model"
   reference_data_path = '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim'
   climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
   #climo_diurnal_subsection = "clim_atm_monthly_diurnal_8xdaily_180x360_aave"
   climo_diurnal_frequency = "diurnal_8xdaily"
   climo_subsection = "atm_monthly_180x360_aave"
   #sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_budget"
    sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","zonal_mean_2d_stratosphere","aerosol_budget"
   diff_title = 'Difference'
   output_format_subplot = "pdf",

   [[ atm_monthly_180x360_aave ]]
 #  environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate edv2110"
   short_name = 'BGCv2LNDATM'
   grid = '180x360_aave'
   reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
   obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
   dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
   climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
   climo_diurnal_frequency = "diurnal_8xdaily"
   ts_daily_subsection = "atm_daily_180x360_aave"
   sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","tropical_subseasonal",
 #"mp_partition","aerosol_budget",
 #  sets="tropical_subseasonal",
   output_format_subplot = "pdf",

   [[ lnd_monthly_mvm_lnd ]]
   # Test model-vs-model using the same files as the reference
   #environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate edv290"
   grid = 'native'
   climo_subsection = "land_monthly_climo"
   diff_title = "Difference"
   partition = "compute"
   qos = "regular"
   short_name = BGCv2LNDATM
   ref_name = "20230717_CBGCv2.r05.chrysalis.I20TRGSWCNP"
   ref_start_yr = 1985
   ref_final_yr = 2014
   ref_years = "1985-2014",
   reference_data_path = "/lcrc/group/e3sm/ac.sfeng1/E3SM_output/20230717_CBGCv2.r05.chrysalis.I20TRGSWCNP/zppy.5yrs/post/lnd/180x360_aave/clim"
   run_type = "model_vs_model"
   sets = "lat_lon_land",
   short_ref_name = "BGCv2LND.GSW"
   swap_test_ref = False
   tag = "model_vs_model"
   ts_num_years_ref = 30

What jobs are failing?

No response

What stack trace are you encountering?

No response

forsyth2 · 2024-06-19T17:59:57Z

forsyth2
Jun 19, 2024
Maintainer

Summary

(scroll left-right)

	Actual	`zppy` errors because it expects	As defined for `[[atm_monthly_180x360_aave_mvm]]` subtask in the given cfg	As defined for the`[[ atm_monthly_diurnal_8xdaily_180x360_aave ]]` subtask in the given cfg
Diurnal files saved under	`clim_atm_monthly_diurnal_8xdaily_180x360_aave`	`clim_diurnal_8xdaily`	`climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"`, `climo_diurnal_frequency = "diurnal_8xdaily"`	N/A
input file name	`eam.h4`	`eam.h3`	`input_files = "eam.h3"`	`input_files = "eam.h3"`

The input file name error is easy then. You need to set input_files = "eam.h4" for the [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] subtask (not the [[atm_monthly_180x360_aave_mvm]] subtask).

As for the the file location, it looks like zppy is using climo_diurnal_frequency rather than climo_diurnal_subsection.

Where is `climo_diurnal_subsection` used?

In https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py:

            if "diurnal_cycle" in c["sets"]:
                if c["dc_obs_climo"] == "":
                    c["dc_obs_climo"] = c["reference_data_path"]
                diurnal_cycle_subtask_name = c["climo_diurnal_subsection"]
                if diurnal_cycle_subtask_name in config["climo"].keys():
                    if (
                        "input_files"
                        in config["climo"][diurnal_cycle_subtask_name].keys()
                    ):
                        diurnal_cycle_input_files = config["climo"][
                            diurnal_cycle_subtask_name
                        ]["input_files"]
                        c["climo_diurnal_input_files"] = diurnal_cycle_input_files

So, climo_diurnal_subsection is used to tell us which climo subtask's input files we should use.

            if "diurnal_cycle" in c["sets"]:
                dependencies.append(
                    os.path.join(
                        scriptDir,
                        "climo_%s_%04d-%04d.status"
                        % (c["climo_diurnal_subsection"], c["year1"], c["year2"]),
                    )
                )

It also is used to tell us which climo subtasks need to finish before diags can run.

Where is `climo_diurnal_frequency` used?

In https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/e3sm_diags.bash:

{% if run_type == "model_vs_obs" %}
climo_diurnal_dir_primary=climo_{{ climo_diurnal_frequency }}
{% elif run_type == "model_vs_model" %}
climo_diurnal_dir_primary=climo_{{ climo_diurnal_frequency }}_test
{%- endif %}
# Create local links to input diurnal cycle climo files
climo_diurnal_dir_source={{ output }}/post/atm/{{ grid }}/clim_{{ climo_diurnal_frequency }}/{{ '%dyr' % (year2-year1+1) }}
create_links_climo_diurnal ${climo_diurnal_dir_source} ${climo_diurnal_dir_primary} ${case} ${Y1} ${Y2} 3

So, climo_diurnal_frequency is used to tell us where to place the links to the diurnal climatology files, as well as where to get the output from the diurnal climatology subtask.

It seems that this may be a bug. Perhaps the line should be this instead:

climo_diurnal_dir_source={{ output }}/post/atm/{{ grid }}/clim_{{ climo_diurnal_subsection }}/{{ '%dyr' % (year2-year1+1) }}

That is, apparently the diurnal climatology subtask outputs files into a directory based on its subsection name (atm_monthly_diurnal_8xdaily_180x360_aave) rather than its frequency parameter (frequency = "diurnal_8xdaily")

0 replies

chengzhuzhang · 2024-06-19T18:36:23Z

chengzhuzhang
Jun 19, 2024
Maintainer Author

@forsyth2 thank you for troubleshooting. I think the complication is that for model vs model case, the test case has diurnal output in h3, however the reference case has output in h4. My understanding is that this configuration file is having difficulty to locate the reference case files which was prepared by another zppy run. The parameters: input_files and climo_diurnal_section/frequency are used for specifying information for the test case, right?

6 replies

forsyth2 Jun 19, 2024
Maintainer

@BunnyVon @chengzhuzhang Can you let me know if setting reference_data_path_climo_diurnal = /lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave for [[atm_monthly_180x360_aave_mvm]] works? The intended use of that parameter is for this situation -- i.e., when zppy can't automatically determine what the path should have been.

I'm doing some testing on a simpler cfg in #608 (I'm trying to create a series of small cfgs to test different zppy options, to replace the one large test that misses many parameter combinations)

chengzhuzhang Jun 19, 2024
Maintainer Author

@forsyth2 I got this error: cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave/30yr/v2.LR.amip_0101._*_1985??_2014??_climo.nc': No such file or directory . reference_data_path_climo_diurnal worked to point to the correct path, but zppy expects eam.h3 and there is not a way to specify a different string eam.h4 for the reference case...

forsyth2 Jun 20, 2024
Maintainer

@chengzhuzhang Thanks for checking! It looks like we'll need to create a new parameter like input_files_ref then...

BunnyVon Jun 20, 2024

Thank you @chengzhuzhang for following this up.

@forsyth2 , I agree that is probably the best solution. I tried to use ref_name = ' v2.LR.amip_0101.eam.h4' to get around it, but got another error cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_diurnal_8xdaily/30yr/v2.LR.amip_0101.eam.h4.eam.h3_*_1985??_2014??_climo.nc': No such file or directory

I think this parameter is quite necessary, particularly for the comparison across different versions and groups that don't have the same output arrangement.

forsyth2 Jun 20, 2024
Maintainer

particularly for the comparison across different versions and groups

Yes, I think the problem is that, at the time of implementation, I thought the h number would be consistent. I've since discovered that is not the case. I should probably do a full review of the parameters being passed to diags to see if anything else needs _ref version too...

forsyth2 · 2024-07-19T22:03:25Z

forsyth2
Jul 19, 2024
Maintainer

This will be resolved by #612.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question on model vs model e3sm_diags task zppy configuration #607

{{title}}

Replies: 3 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Question on model vs model e3sm_diags task zppy configuration #607

chengzhuzhang Jun 19, 2024 Maintainer

Question criteria

What is the deadline?

Describe your question

Are there are any possible answers you came across?

What machine were you running on?

Environment

What command did you run?

Copy your cfg file

What jobs are failing?

What stack trace are you encountering?

Replies: 3 comments · 6 replies

forsyth2 Jun 19, 2024 Maintainer

Summary

Where is climo_diurnal_subsection used?

Where is climo_diurnal_frequency used?

chengzhuzhang Jun 19, 2024 Maintainer Author

forsyth2 Jun 19, 2024 Maintainer

chengzhuzhang Jun 19, 2024 Maintainer Author

forsyth2 Jun 20, 2024 Maintainer

BunnyVon Jun 20, 2024

forsyth2 Jun 20, 2024 Maintainer

forsyth2 Jul 19, 2024 Maintainer

chengzhuzhang
Jun 19, 2024
Maintainer

Replies: 3 comments 6 replies

forsyth2
Jun 19, 2024
Maintainer

Where is `climo_diurnal_subsection` used?

Where is `climo_diurnal_frequency` used?

chengzhuzhang
Jun 19, 2024
Maintainer Author

forsyth2 Jun 19, 2024
Maintainer

chengzhuzhang Jun 19, 2024
Maintainer Author

forsyth2 Jun 20, 2024
Maintainer

forsyth2 Jun 20, 2024
Maintainer

forsyth2
Jul 19, 2024
Maintainer