Dynamic Smagorinsky on HPC

[SamDeAbreu]

Hi all,

I see that there has been development by @tomchor and others on a Smagorinsky closure but with a dynamic coefficient, which is amazing! I've been able to run the directionally-averaged version locally, but run into issues when running it on my school's HPC (version 0.95.10). Specifically, the code stalls on building the nonhydrostatic model.

It appears this is a known issue (#3642 (comment)) when it was being tested on a server. I was wondering if the cause has been determined yet? Or if we're still not sure?

Thanks,
Sam

[tomchor]

I think we never understood why that compilation wasn't working on sverdrup. When we switched to using Github's machines (#3926) it worked pretty well, so I think compiling the dynamic coefficient is just too much for some machines... ?

That said, it takes a looong time to compile. How long have you waited for before throwing in the towel?

[SamDeAbreu]

I've waited up to 30min trying to build a 64x64x64 model, but the HPC kills it/runs out of memory before anything else happens. Very weird that I can build the 64x64x64 model on my okay local GPU in a time <5min, but not on a more advanced GPU on an HPC.