Skip to content

Make domain configuration file

Jump to bottom
davbyr edited this page May 25, 2022 · 3 revisions

To create a new configuration a domain configuration file must be created that defines the grid. This only needs to be done once for a new configuration. This includes bathymetry, depth and horizontal grid discretisation.

Gather files

Code

The following is written with ARCHER2 in mind:

cd /work/n01/n01/$USER
git clone https://github.com/JMMP-Group/CO_AMM15.git
cd /work/n01/n01/$USER/CO9_AMM15
git checkout ME_domaincfg # only if it hasn't been merged or renamed...

Coordinates

Start with an AMM15 coordinates file. This stores the horizontal grid information. This is copied to 'DOWNLOADS' for processing. E.g.

For example on ARCHER2:

mkdir DOWNLOADS
cp /work/n01/n01/shared/dbyrne/AMM15_ME/INPUTS/amm15.coordinates.nc /work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/.
#scp /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/coordinates.nc [email protected]:/work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/coordinates_AMM15.nc
#scp /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/coordinates.nc archer2:/work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/coordinates_AMM15.nc

bathymetry

This is copied to 'DOWNLOADS' for processing. E.g.

cp /work/n01/n01/shared/dbyrne/AMM15_ME/INPUTS/amm15.bathydepth.hook.nc /work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/.
#scp /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/amm15_bathy.nc [email protected]:/work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/.
#scp /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/amm15_bathy.nc archer2:/work/n01/n01/$USER/CO9_AMM15/DOWNLOADS/.

Preprocess the bathymetry, to include envelopes

minimum depth 10m

Then if wetting and drying is not used, set a minimum depth of 10m. Actually this can be controlled by the namelist_cfg in the make_domain_cfg.exe step. See rn_bot_min in &namzgr_mes. Anyway, this is how to do it in python:

  cd DOWNLOADS

  # load nco modules (this was done locally)
  module load nco/4.9.6 #otherwise ncap2 not available

  # pull out the bathymetry variable as we don't need coordinate variables
  ncks -C -v Bathymetry amm15.bathydepth.hook.nc amm15.bathydepth.nc

  # Remove weirdness with negative bathymetry and make minimum bathymetry
  #equal to 10 m (resolve any possible wet-drying problems)
  ncap2 -s 'where(Bathymetry < 0) Bathymetry=0' amm15.bathydepth.hook.nc tmp1.nc
  ncap2 -s 'where(Bathymetry < 10 && Bathymetry > 0) Bathymetry=10' tmp1.nc -O amm15.10m.bathy_meter.nc
  rm tmp1.nc

  # Relabel the dimensions (there are issues with using lat and lon and dimension names with the envelope generating script)
  ncrename -d lon,x -d lat,y amm15.10m.bathy_meter.nc tmp2.nc
  mv tmp2.nc amm15.10m.bathy_meter.nc

WARNING It is possible that this 10m preprocessing messed stuff up. Diego reported MPI errors on MONSooN when running make_domain_cfg.exe after having done these steps but things being OK when taking a short cut to sort out dimension names only with:

ncks -C -v Bathymetry amm15.bathydepth.hook.nc amm15.10m.bathy_meter.nc
# target filename preserved with "10m" for consistency with `$BATHYFILE` in the following steps.

Embed envelope bathymetry into bathy file

A python tool, generate_envelopes.py, is used to create the envelope bathymetry variables and embed them into the bathymetry file. This tool is controlled by an input file. Insert the bathymetry and coordinates file names into the input file. (I'm anticipating scripting this step, so these names are already defined in scripts/make_domain_cfg_AMM15.sh):

The paths are cut and paste here for reference (but should be deleted ASAP or they will fall out of sync):

## Make paths
################################
export WORK=/work/n01/n01/$USER
export WDIR=$WORK/CO9_AMM15 # Also the git clone directory
export NEMO_VER=4.0.4
export NEMO=$WDIR/BUILD_CFG/$NEMO_VER

export TDIR=$NEMO/tools
export DOMAIN=$WDIR/BUILD_CFG/DOMAIN
export DOWNLOADS=$WDIR/DOWNLOADS

export BATHYFILE=$DOWNLOADS/amm15.10m.bathy_meter.nc
export HGRIDFILE=$DOWNLOADS/amm15.coordinates.nc

sed "s?XXX_BATHYFILE_XXX?$BATHYFILE?" $DOMAIN/../MEs_envs/MEs_2env_0.1_0.07_opt_template.inp > $DOMAIN/../MEs_envs/tmp1.inp
sed "s?XXX_HGRIDFILE_XXX?$HGRIDFILE?" $DOMAIN/../MEs_envs/tmp1.inp > $DOMAIN/../MEs_envs/MEs_2env_0.1_0.07_opt.inp
rm $DOMAIN/../MEs_envs/tmp1.inp

Next run the python generate_envelopes.py tool to create the envelope bathymetry variables and embed them into the bathymetry file. On ARCHER2 this required a new conda environment.

# matplotlib was missing from my basic python env, so created a fresh one:
conda create --name me_envs numpy matplotlib xarray netCDF4
conda activate me_envs
cd BUILD_CFG/MEs_envs
python generate_envelopes.py MEs_2env_0.1_0.07_opt.inp

This generates bathymetry.MEs_2env_0.1_0.07_opt.nc, a bathymetry with envelopes hbatt_1 and hbatt_2 within.

Make the domain configuration file

Generate the domain_cfg.nc file using the coordinates.nc, bathy_meter.nc (symbolic links are used to standardise these file names). This is done in the tools directory using make_domain_cfg_AMM15.sh script in the SCRIPTS directory. The tools used are the DOMAINcfg and REBUILD_NEMO tools. The DOMAINcfg tool is submitted as a slurm job.

If making changes to the configuration, these are controlled by the namelist_cfg file. Ensure this has the appropriate parameters and number of lat,lon,depth levels etc set.

For example the lateral size of the domain can be extracted from bathy_meter.nc using ncdump -h bathy_meter.nc, then namelist can be edited accordingly e.g.:

vi namelist_cfg
   ...
   jpidta      =     1458   !  1st lateral dimension ( >= jpi )
   jpjdta      =     1345   !  2nd    "         "    ( >= jpj )
   jpkdta      =       51   !  number of levels      ( >= jpk )
   ...

When ready, run:

cd scripts
. ./make_domain_cfg_AMM15.sh

creating $DOMAIN/domain_cfg_AMM15.nc

This script has a number of steps:

  • sets some paths
  • obtains the NEMO and tools code (uses MO branch with Diego's ME tool)
  • Compiles the tools on ARCHER2
  • submits a job to create the domain_cfg.nc on ARCHER2 (users might need to change the billing account)

Wetting & Drying

If making a domain_cfg file for use with Enda's wetting and drying scheme, there will need to be two extra variables present as well as some tweaks to the domainCFG tool. This is the native v4.0 tool that comes packaged with NEMO in the /tools directory. The modifications are in the BUILD_CFG/domwad directory of this repo. There are 3 files modified here:

  1. dom_oce.F90
  2. domain.F90
  3. domzgr.F90

Together, these modifications do the following:

  1. Read in the rn_hc variable from namelist_cfg (see example in /domwad) and add this value to the bathymetry prior to generating the domain. This is a necessary step in the case of ln_sco=True.
  2. Output rn_wd_ref_depth to domain_cfg.nc. This is necessary for the wetting and drying.
  3. Output ht_wd to domain_cfg.nc. Also needed for wetting and drying.

The first point above only works in the case of ln_sco. Otherwise, nothing will be different. For ln_mes, points 2 and 3 still work so the domain file should work with a NEMO run including WAD. No changes are made to the bathymetry and it is still unclear if there needs to be any. If so, then generate_envelopes.py is the place for it to happen.

To use these modifications, simple download NEMO/tools and copy them into DOMAINcfg/src before compiling.

Clone this wiki locally