How do I get the lfric_apps azure spice rose-stem to work?

[MichaelWhitall]

Hi, me again!

I have a ticket:
https://code.metoffice.gov.uk/trac/lfric_apps/ticket/481
where I've updated the comorph_dev rose-stem app and therefore need to update its KGO checksums on all platforms. At some point someone very diligently added copies of the comorph SCM rose-stem tests to the new azure-spice test group. So I need to run the azure spice tests to generate the updated KGOs. I've not yet run the azure spice rose stem group before. Running rose stem from my lfric_apps test branch (on my old VDI) with --group=azspice_developer, it all submits to azure spice fine (I have done the ssh setup etc), but most of the compiles and a few other things are failing with:

ModuleCmd_Use.c(231):ERROR:64: Directory '/home/users/lfricadmin/lmod' not found
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'lfric/vn2.0'

Maybe there is some more setup I need to do on azure spice before I can submit rose stem tests to it? Or does it only work if I run rose stem from a working copy on azure spice? The above error suggests that a whole directory it expects to find under /home/users doesn't exist, so I can't see how to fix this. Any ideas?

Cheers!
Mike

[james-bruten-mo]

Hi Mike, Yep, you can only launch on azspice when launching from azspice. Similarly you can't launch on old spice from azspice. Thanks for taking the time to get set up and start testing there!

[MichaelWhitall]

OK, cool thanks...

What's the agreed working practice then, for updating KGOs in an lfric_apps ticket which changes answers on both old and new spice? (since this state-of-affairs means it is impossible to actually run all the tests in a single submission of rose-stem?) Do we literally need to have 2 separate working copies of the same ticket's branch on VDI and azure-spice, run the rose-stem tests separately from each platform, then update and commit the change to the KGO checksums separately on each? (remembering to fcm update the working copy on the latter after committing the update from the former?)

I think given how short we are for time now before the code deadline, is it OK if I sort out the azure spice testing / KGO-update for my various tickets after submission to code review?

Cheers!
Mike

[stevemullerworth]

I posted my recommendations below at about the same time you asked this!

[MichaelWhitall]

Yep, thanks Steve, yes I think we both posted at the same time there ;)

So the upshot is, yes the azure spice rose stem tests ought to be done before review, and yes this does require maintaining separate working copies of branches on both the old and new platforms in order to run all the required tests (for the record I was all-for running tests on all 4 platforms before review, but I naively thought I could submit them all from a single working copy on old VDI).

I've now managed to get MOSRS access and fcm setup on azure-spice and my first round of lfric_apps rose-stem tests are now running from there :) Encouragingly, I swear the cylc 8 gui is running a lot less laggy on azure-spice than it does on old VDI; if this bares-out, I'm a convert!

[stevemullerworth]

At the Met Office:
Running developer tests from old spice submits them to old spice and the XC machines.
Running developer tests from new (AZ Spice) submits them to azspice and the EX machines (EXA and EXB).

Currently, we strongly encourage testing that your jobs run on all four platforms. Testing LFRic on the new platforms has exposed some bugs in both LFRic and UM code, so it would be best to pick them up before review.

As yet, science is not being formally reviewed on the new platforms (and not all tests have been ported to the new platforms). While we are keeping KGOs up to date on trunk, they do not necessarily reflect good scientific results. Generating correct KGOs is done at the code review and commit stage, so if you submit a ticket without having updated your AZSpice/EX KGOs then your reviewer is not likely to be concerned as long as you have run the tests.

That situation will change as soon as we move towards accepting the new platforms or as soon as the relevant scientists have positively assessed the test results on the new platforms, so please pay attention to updates!

[MichaelWhitall]

Hi Steve,

You'll be pleased to hear my attempt to test my lfric_apps branch on azure-spice has in fact dredged up a bug in the existing lfric_apps code, so I'm working on fixing that :) As far as I can tell the bug is not related to my branch; there's an existing unassociated pointer flux_e passed into a subroutine in bl_exp_alg_mod.x90.

But that aside, I can't get the ex1a rose stem tests to work when launching from azure-spice. The first few things submit fine, but then all the ex1a build_lfric_atm tasks just hang in the queue indefinitely and eventually go into a submit-fail state. Does anyone know what I need to do to get azure spice to get the compile jobs to submit to ex1a? Or should I just keep launching my ex1a tests from old VDI for now? (that's been working fine!)

Cheers!
Mike

[MichaelWhitall]

Update: I managed to get azure spice to successfully run ex1a rose stem tests for lfric_apps in the end. It turned out I needed to also setup passphraseless ssh from ex1a to azure spice. I had a recollection that unlike the UM, the lfric build only required ssh in one direction (from the platform where you launch rose-stem to the platform where the compile happens), but this doesn't seem to be the case; it only worked after I setup the ssh-keys for the reverse as well (not that the error messages gave me any clues at all Re what the problem was!)

[stevemullerworth]

The flux_e issue was fixed on trunk by the following ticket:

https://code.metoffice.gov.uk/trac/lfric_apps/ticket/512