Merge pull request #1714 from slevisconsulting/reduce_redundancy

Reduce redundancy; make interfaces more similar between tools; focusing here on fsurdat_modifier and subset_data

.git-blame-ignore-revs

@@ -2,3 +2,4 @@
 4cd83cb3ee6d85eb909403487abf5eeaf4d98911
 0aa2957c1f8603c63fa30b11295c06cfddff44a5
 2cdb380febb274478e84cd90945aee93f29fa2e6
+51c102c5df2e0ef971b5f8eeeb477567899af63a

README

@@ -45,7 +45,7 @@ test -------------- CTSM Testing scripts for CTSM offline tools.
 tools ------------- CTSM Offline tools to prepare input datasets and process output.
 cime_config ------- Configuration files of cime for compsets and CTSM settings
 manage_externals -- Script to manage the external source directories
-manage_python_env - Script to setup the python environment for CTSM python tools using conda
+py_env_create ----- Script to setup the python environment for CTSM python tools using conda
 python ------------ Some python modules mostly for use by run_sys_tests (but could be used elsewhere l
 
 Directory structure only for a CTSM checkout:

doc/ChangeLog

@@ -1,4 +1,71 @@
 ===============================================================
+Tag name: ctsm5.1.dev105
+Originator(s): slevis (Samuel Levis,SLevis Consulting,303-665-1310)
+Date: Tue Jul 26 23:57:31 MDT 2022
+One-line Summary: make interfaces more similar between fsurdat_modifier and subset_data tools
+
+Purpose and description of changes
+----------------------------------
+
+ The tools modify_fsurdat and subset_data have similar options
+ modify_fsurdat: zero_nonveg
+ subset_data: include-nonveg
+
+ modify_fsurdat: dom_plant
+ subset_data: dom-pft
+
+ Make more similar and use README file for documenting modify_fsurdat.
+ There's potential for additional changes to be addressed later.
+
+ Change name of manage_python_env to py_env_create as agreed in CTSM software meeting
+
+Significant changes to scientifically-supported configurations
+--------------------------------------------------------------
+
+Does this tag change answers significantly for any of the following physics configurations?
+(Details of any changes will be given in the "Answer changes" section below.)
+
+    [Put an [X] in the box for any configuration with significant answer changes.]
+
+[ ] clm5_1
+
+[ ] clm5_0
+
+[ ] ctsm5_0-nwp
+
+[ ] clm4_5
+
+
+Bugs fixed or introduced
+------------------------
+CTSM issues fixed (include CTSM Issue #): #1662
+
+Testing summary:
+----------------
+
+ [PASS means all tests PASS; OK means tests PASS other than expected fails.]
+
+  python testing (if python code has changed; see instructions in python/README.md; document testing done):
+
+    cheyenne - PASS
+
+    clm_pymods test suite on cheyenne - PASS
+
+If the tag used for baseline comparisons was NOT the previous tag, note that here: 
+
+
+Answer changes
+--------------
+Changes answers relative to baseline: No
+
+Other details
+-------------
+Pull Requests that document the changes (include PR ids):
+ https://github.com/ESCOMP/ctsm/pull/1714
+ https://github.com/ESCOMP/ctsm/pull/1799
+
+===============================================================
+===============================================================
 Tag name: ctsm5.1.dev104
 Originator(s): glemieux (Gregory Lemieux,LBL/NGEET,510-486-5049)
 Date: Mon Jul 18 15:45:00 MDT 2022

doc/ChangeSum

@@ -1,5 +1,6 @@
 Tag                   Who      Date  Summary
 ============================================================================================================================
+    ctsm5.1.dev105   slevis 07/26/2022 make interfaces more similar between fsurdat_modifier and subset_data tools
     ctsm5.1.dev104 glemieux 07/15/2022 Update to FATES history names and machine configuration
     ctsm5.1.dev103    sacks 07/12/2022 Fix accumulation variables when changing model time step
     ctsm5.1.dev102    sacks 07/11/2022 Fix LILAC interface to PIO

manage_python_env → py_env_create

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# manage_python_env -- setup the python environment in order to use CTSM python tools
+# py_env_create -- setup the python environment in order to use CTSM python tools
 #
 # Simple bash script to setup the python environment for the user so they can run the CTSM
 # python tools using "conda".
@@ -40,7 +40,7 @@ usage() {
   echo ""
   echo "***********************************************************************"
   echo "usage:"
-  echo "./manage_python_env"
+  echo "./py_env_create"
   echo ""
   echo "valid arguments: "
   echo "[-h|--help]  "

python/README.md

@@ -13,7 +13,7 @@ a python environment. Comments in the file tell how to do this.
 You can also use the script in the top level directory to do
 all the conda commands and do this for you.
 
- ../manage_python_env
+ ../py_env_create
  conda activate ctsm_py
 
 Conda requirements files:

python/conda_env_ctsm_py.txt

@@ -3,7 +3,7 @@
 #
 # To install this on cheyenne with conda loaded in modules
 # use the top level bash script:
-# ../manage_python_env                                    # Do this each time you update your CTSM Version
+# ../py_env_create                                        # Do this each time you update your CTSM Version
 # conda activate ctsm_py                                  # Do this anytime you want to run a CTSM python script
 # Or the individual conda commands:
 # conda create  -n ctsm_py                                # Do this one time for each machine

python/conda_env_ctsm_py_cgd.txt

@@ -5,7 +5,7 @@
 #
 # To install this on cheyenne with conda loaded in modules
 # use the top level bash script:
-# ../manage_python_env                                    # Do this each time you update your CTSM Version
+# ../py_env_create                                        # Do this each time you update your CTSM Version
 # conda activate ctsm_py                                  # Do this anytime you want to run a CTSM python script
 # Or the individual conda commands:
 # conda create  -n ctsm_py                                # Do this one time for each machine

python/ctsm/modify_fsurdat/fsurdat_modifier.py

@@ -62,10 +62,10 @@ def fsurdat_modifier(cfg_path):
         file_path=cfg_path,
         convert_to_type=bool,
     )
-    zero_nonveg = get_config_value(
+    include_nonveg = get_config_value(
         config=config,
         section=section,
-        item="zero_nonveg",
+        item="include_nonveg",
         file_path=cfg_path,
         convert_to_type=bool,
     )
@@ -114,10 +114,10 @@ def fsurdat_modifier(cfg_path):
 
     # not required: user may set these in the .cfg file
     max_pft = int(max(modify_fsurdat.file.lsmpft))
-    dom_plant = get_config_value(
+    dom_pft = get_config_value(
         config=config,
         section=section,
-        item="dom_plant",
+        item="dom_pft",
         file_path=cfg_path,
         allowed_values=range(max_pft + 1),  # integers from 0 to max_pft
         convert_to_type=int,
@@ -201,7 +201,7 @@ def fsurdat_modifier(cfg_path):
     if idealized:
         modify_fsurdat.set_idealized()  # set 2D variables
         # set 3D and 4D variables pertaining to natural vegetation
-        modify_fsurdat.set_dom_plant(dom_plant=0, lai=[], sai=[], hgt_top=[], hgt_bot=[])
+        modify_fsurdat.set_dom_pft(dom_pft=0, lai=[], sai=[], hgt_top=[], hgt_bot=[])
         logger.info("idealized complete")
 
     if max_sat_area is not None:  # overwrite "idealized" value
@@ -216,17 +216,17 @@ def fsurdat_modifier(cfg_path):
         modify_fsurdat.setvar_lev0("SOIL_COLOR", soil_color)
         logger.info("soil_color complete")
 
-    if zero_nonveg:
+    if not include_nonveg:
         modify_fsurdat.zero_nonveg()
         logger.info("zero_nonveg complete")
 
-    # The set_dom_plant call follows zero_nonveg because it modifies PCT_NATVEG
+    # set_dom_pft follows zero_nonveg because it modifies PCT_NATVEG
     # and PCT_CROP in the user-defined rectangle
-    if dom_plant is not None:
-        modify_fsurdat.set_dom_plant(
-            dom_plant=dom_plant, lai=lai, sai=sai, hgt_top=hgt_top, hgt_bot=hgt_bot
+    if dom_pft is not None:
+        modify_fsurdat.set_dom_pft(
+            dom_pft=dom_pft, lai=lai, sai=sai, hgt_top=hgt_top, hgt_bot=hgt_bot
         )
-        logger.info("dom_plant complete")
+        logger.info("dom_pft complete")
 
     # ----------------------------------------------
     # Output the now modified CTSM surface data file

python/ctsm/modify_fsurdat/modify_fsurdat.py

@@ -134,34 +134,34 @@ def write_output(self, fsurdat_in, fsurdat_out):
         logger.info("Successfully created fsurdat_out: %s", fsurdat_out)
         self.file.close()
 
-    def set_dom_plant(self, dom_plant, lai, sai, hgt_top, hgt_bot):
+    def set_dom_pft(self, dom_pft, lai, sai, hgt_top, hgt_bot):
         """
         Description
         -----------
         In rectangle selected by user (or default -90 to 90 and 0 to 360),
         replace fsurdat file's PCT_NAT_PFT or PCT_CFT with:
-        - 100 for dom_plant selected by user
+        - 100 for dom_pft selected by user
         - 0 for all other PFTs/CFTs
         If user has specified lai, sai, hgt_top, hgt_bot, replace these with
-        values selected by the user for dom_plant
+        values selected by the user for dom_pft
 
         Arguments
         ---------
-        dom_plant:
+        dom_pft:
             (int) User's entry of PFT/CFT to be set to 100% everywhere
         lai:
-            (float) User's entry of MONTHLY_LAI for their dom_plant
+            (float) User's entry of MONTHLY_LAI for their dom_pft
         sai:
-            (float) User's entry of MONTHLY_SAI for their dom_plant
+            (float) User's entry of MONTHLY_SAI for their dom_pft
         hgt_top:
-            (float) User's entry of MONTHLY_HEIGHT_TOP for their dom_plant
+            (float) User's entry of MONTHLY_HEIGHT_TOP for their dom_pft
         hgt_bot:
-            (float) User's entry of MONTHLY_HEIGHT_BOT for their dom_plant
+            (float) User's entry of MONTHLY_HEIGHT_BOT for their dom_pft
         """
 
-        # If dom_plant is a cft, add PCT_NATVEG to PCT_CROP in the rectangle
+        # If dom_pft is a cft, add PCT_NATVEG to PCT_CROP in the rectangle
         # and remove same from PCT_NATVEG, i.e. set PCT_NATVEG = 0.
-        if dom_plant > max(self.file.natpft):  # dom_plant is a cft (crop)
+        if dom_pft > max(self.file.natpft):  # dom_pft is a cft (crop)
             self.file["PCT_CROP"] = self.file["PCT_CROP"] + self.file["PCT_NATVEG"].where(
                 self.rectangle, other=0
             )
@@ -173,13 +173,13 @@ def set_dom_plant(self, dom_plant, lai, sai, hgt_top, hgt_bot):
                 self.setvar_lev1("PCT_CFT", val=0, lev1_dim=cft_local)
 
             # set 3D variable
-            self.setvar_lev1("PCT_CFT", val=100, lev1_dim=dom_plant - (max(self.file.natpft) + 1))
-        else:  # dom_plant is a pft (not a crop)
+            self.setvar_lev1("PCT_CFT", val=100, lev1_dim=dom_pft - (max(self.file.natpft) + 1))
+        else:  # dom_pft is a pft (not a crop)
             for pft in self.file.natpft:
                 # initialize 3D variable; set outside the loop below
                 self.setvar_lev1("PCT_NAT_PFT", val=0, lev1_dim=pft)
-            # set 3D variable value for dom_plant
-            self.setvar_lev1("PCT_NAT_PFT", val=100, lev1_dim=dom_plant)
+            # set 3D variable value for dom_pft
+            self.setvar_lev1("PCT_NAT_PFT", val=100, lev1_dim=dom_pft)
 
         # dictionary of 4d variables to loop over
         vars_4d = {
@@ -190,17 +190,17 @@ def set_dom_plant(self, dom_plant, lai, sai, hgt_top, hgt_bot):
         }
         for var, val in vars_4d.items():
             if val is not None:
-                self.set_lai_sai_hgts(dom_plant=dom_plant, var=var, val=val)
+                self.set_lai_sai_hgts(dom_pft=dom_pft, var=var, val=val)
 
-    def set_lai_sai_hgts(self, dom_plant, var, val):
+    def set_lai_sai_hgts(self, dom_pft, var, val):
         """
         Description
         -----------
         If user has specified lai, sai, hgt_top, hgt_bot, replace these with
-        values selected by the user for dom_plant. Else do nothing.
+        values selected by the user for dom_pft. Else do nothing.
         """
         months = int(max(self.file.time))  # 12 months
-        if dom_plant == 0:  # bare soil: var must equal 0
+        if dom_pft == 0:  # bare soil: var must equal 0
             val = [0] * months
         if len(val) != months:
             errmsg = (
@@ -211,8 +211,8 @@ def set_lai_sai_hgts(self, dom_plant, var, val):
             )
             abort(errmsg)
         for mon in self.file.time - 1:  # loop over 12 months
-            # set 4D variable to value for dom_plant
-            self.setvar_lev2(var, val[int(mon)], lev1_dim=dom_plant, lev2_dim=mon)
+            # set 4D variable to value for dom_pft
+            self.setvar_lev2(var, val[int(mon)], lev1_dim=dom_pft, lev2_dim=mon)
 
     def zero_nonveg(self):
         """

python/ctsm/site_and_regional/single_point_case.py

@@ -56,8 +56,6 @@ class SinglePointCase(BaseCase):
         weight or percentage of each pft.
     num_pft : list
         total number of pfts for surface dataset (if crop 78 pft, else 16 pft)
-    zero_nonveg_landunits : bool
-        flag for setting all non-vegetation landunits to zero
     uni_snow : bool
         flag for creating datasets using uniform snowpack
     saturation_excess : bool

python/ctsm/test/test_sys_fsurdat_modifier.py

@@ -157,8 +157,8 @@ def _create_config_file_crop(self):
                         line = "lnd_lon_1 = 295\n"
                     elif re.match(r" *lnd_lon_2 *=", line):
                         line = "lnd_lon_2 = 300\n"
-                    elif re.match(r" *dom_plant *=", line):
-                        line = "dom_plant = 15"
+                    elif re.match(r" *dom_pft *=", line):
+                        line = "dom_pft = 15"
                     elif re.match(r" *lai *=", line):
                         line = "lai = 0 1 2 3 4 5 5 4 3 2 1 0\n"
                     elif re.match(r" *sai *=", line):
@@ -192,8 +192,8 @@ def _create_config_file_complete(self):
                         line = "lnd_lon_1 = 295\n"
                     elif re.match(r" *lnd_lon_2 *=", line):
                         line = "lnd_lon_2 = 300\n"
-                    elif re.match(r" *dom_plant *=", line):
-                        line = "dom_plant = 1"
+                    elif re.match(r" *dom_pft *=", line):
+                        line = "dom_pft = 1"
                     elif re.match(r" *lai *=", line):
                         line = "lai = 0 1 2 3 4 5 5 4 3 2 1 0\n"
                     elif re.match(r" *sai *=", line):

test/tools/test_driver.sh

@@ -54,7 +54,7 @@ module load nco
 module load ncl
 
 module load conda
-$CESMDATAROOT/manage_python_env
+$CESMDATAROOT/py_env_create
 conda activate ctsm_py
 
 
@@ -119,7 +119,7 @@ module load openmpi
 module load nco
 module load conda
 module load ncl
-$CESMDATAROOT/manage_python_env
+$CESMDATAROOT/py_env_create
 conda activate ctsm_py
 
 
@@ -220,7 +220,7 @@ module load compiler/intel
 module load tool/nco
 module load tool/netcdf
 module load lang/python
-$CESMDATAROOT/manage_python_env
+$CESMDATAROOT/py_env_create
 conda activate ctsm_py
 
 export NETCDF_DIR=\$NETCDF_PATH
@@ -303,7 +303,7 @@ module load compiler/intel
 module load tool/nco
 module load tool/netcdf
 module load lang/python
-$CESMDATAROOT/manage_python_env
+$CESMDATAROOT/py_env_create
 conda activate ctsm_py
 
 export NETCDF_DIR=\$NETCDF_PATH

tools/README

@@ -19,6 +19,8 @@ I.  General directory structure:
                            standard process as well as subsetting existing datasets and overwriting
                            some aspects for a specific case.
 
+        modify_fsurdat --- Modify fsurdat file with idealized/simplified settings
+
         contrib ---------- Miscellaneous tools for pre or post processing of CTSM.
                            Typically these are contributed by anyone who has something
                            they think might be helpful to the community. They may not

tools/contrib/README

@@ -9,8 +9,7 @@ assume NCAR directory structures.
 The python scripts require the following settings before running on cheyenne:
 
 module load conda
-cd ../..
-./manage_python_env
+../../py_env_create
 conda activate ctsm_py
 
 Brief description of scripts:

tools/contrib/ssp_anomaly_forcing_smooth

@@ -7,8 +7,8 @@ Create anomoly forcing datasets for SSP scenarios that can be used by CESM datm
 
 load proper modules first, i.e.
 
+../../py_env_create
 conda activate ctsm_py
-(use ../../manage_python_env)
 
 """
 import sys