ESMValTool Killed when increasing amount of input data

[morobking]

I've been writing a diagnostic looking at LST and Air Temperature in CMIP6 models.
While writing and testing, I've been using a 3 year period of data and 2 or 3 models.
I now want the results, from 10 or more models, from a 12 year period.
But while ESMValTool is loading the data/ creating tasks from the recipe, it fails with 'Killed' as the the only messgae on the command line. It has printed to screen before 100s of messages that are things like filenames being loaded, 'adding bounds' to data etc which normally happen.

Is there something I can change in how I'm running ESMValTool so it can handle this amount of data?
I'm currently running on Jasmin from the interactive Linux command line.

This is for branch cmug_wp4.9
with recipe: esmvaltool/recipes/recipe_cmug_wp4_9.yml
and diagnostic: esmvaltool/diag_scripts/cmug_wp4_9/global.py

[zklaus]

Could you try submitting this run to a queue and share with us the resulting slurm-XYZ.out file?

[morobking]

I'm just trying running via SLURM.
Do you have any estimates of time and memory I should be requesting?

[zklaus]

Sorry, I don't, though I suspect that your problem on the interactive node was excessive use of memory.

[morobking]

This is from the the slurm**.out file for 2 CMIP6 models, 11 years + 2 'obs' datasets

2022-05-24 13:52:09,899 UTC [99647] INFO Time for running the recipe was: 0:46:40.030563
2022-05-24 13:52:10,887 UTC [99647] INFO Maximum memory used (estimate): 22.5 GB

and this is the same but for 3 CMIP6 models:
2022-05-24 14:53:29,750 UTC [277413] INFO Time for running the recipe was: 0:46:13.732733
2022-05-24 14:53:29,837 UTC [277413] INFO Maximum memory used (estimate): 21.9 GB

and 4 CMIP models:

I have set this in the batch file:
2022-05-24 15:53:03,854 UTC [158430] INFO Time for running the recipe was: 0:58:53.390351
2022-05-24 15:53:03,970 UTC [158430] INFO Maximum memory used (estimate): 22.4 GB

#SBATCH --time=90:00
#SBATCH --mem=50000
#SBATCH --ntasks=4

So memory on the interactive node looks like the issue - first time I've had that hence not having explored SLURM before.
If I've understood the SLURM documentation correct, --men=50000 is a request for 50Gb?
And from these examples, to me it looks like ESMValTool won't use much more memory for adding a single extra CMIP model but that does increase the run time.
Is that expected behaviour around memory?

[zklaus]

I think this may make sense as the most memory is usually used in regridding. This does not need memory for all models at the same time, so the total max memory is likely determined by the single model that needs that amount of memory and all others fall comfortably in that envelope. The memory needed is determined by the number of cells in the input, the number of cells in the output, and possibly by the overlap pattern of the two grids.

The problem may be exasperated by the interactive machines, because here you have to share the memory with all other users that currently are logged in to that machine. So even though they have large amounts of memory, it is never quite predictable how much of that may be available for you. On the batch node, on the other hand, you don't have to share the memory with anyone else, so it is much more predictable. Besides, it is generally a better usage pattern to allow scalability under the watchful eyes of the HPC admins.

[morobking]

I've got ESMValTool running via SLURM on Jasmin.
This is the first time I've needed more resource than what is available on the interactive nodes.

In the SLURM-***.out files, I'm seeing for 4 CMIP models, and 2 obs

2022-05-24 15:53:03,854 UTC [158430] INFO Time for running the recipe was: 0:58:53.390351
2022-05-24 15:53:03,970 UTC [158430] INFO Maximum memory used (estimate): 22.4 GB

and for 5 CMIP modes

2022-05-24 16:15:16,280 UTC [192906] INFO Time for running the recipe was: 1:14:22.773929
2022-05-24 16:15:16,899 UTC [192906] INFO Maximum memory used (estimate): 22.2 GB

I've been requesting this resouce
#SBATCH --time=90:00
#SBATCH --mem=50000
#SBATCH --ntasks=4

So I agree @zklaus this looks like an excessive use of memory problem.
And hopefully, when I've just finished coding the last plot I can ramp up to larger numbers of CMIP models, requesting 2-3 hours of time and the same amount of memory.

[morobking]

Thanks @zklaus this is all sorted by using batch nodes with more memory.

[RChapman1407]

Could someone please direct me to some more detailed documentation on running ESMValtool via SLURM? This doesn't seem very clear from the JASMIN documentation. (I'm also happy to open a new discussion if that would be more appropriate.)
Thank you

[zklaus]

Thanks for the suggestion, @RChapman1407, and sorry for the long delay (holidays...). Yes, I think it would be good to open something new, in this case best an issue, I think, since this is an enhancement request for the documentation. Are you up for opening that issue? If so, please do.