Identify module variables used by kernels

[arporter]

In OpenCL (#174) and OpenACC we have a problem because we currently allow a Fortran kernel to use data from modules other than the one containing it. In OpenCL, the only way of providing data to a kernel is via an argument. In OpenACC, the original declaration of any module data used'd by a kernel must be marked-up with a directive. Therefore in both cases we are going to need to parse the kernel and identify any module data that is imported.

[arporter]

In both cases, we could solve the problem by re-writing the kernel so that the module data is passed by argument instead. The module 'use' can then be promoted into the PSy layer. This transformation will require the fparser2 AST of the kernel which is (part of) the subject of #185.

[arporter]

However, if we pass the module data by argument, we will need then need to add declarations for those arguments within the kernel subroutine. This will require us to find the original declarations of the module data (by parsing module code).

[arporter]

For OpenACC it is therefore probably simplest to modify the module code by the insertion of the necessary directive (to instruct the compiler that the quantity being declared is required on the device).

[stevemullerworth]

Couple of questions:

• how will you find the module code? It's not currently all in one place in the LFRic repository. Do we need to think about organising it so it can be more easily discoverable.
• currently, kernels can use configuration information obtained from Fortran namelists. These are read into the application with code that is itself autogenerated by our "configurator" application. So we may end up with two lots of autogeneration...I guess that works but is it feasible to fold the requirement into the configurator instead?

[arporter]

For the 1st question we will let the user specify one or more search paths on the command line. Possibly in the same way as we do for Fortran include files. Therefore as long as the build system knows where the modules live we should be fine.
The 2nd question is not so easy to answer. Do the kernels access the configuration information via standard 'use' statements but the modules which are use'd are auto-generated? If so we need to stick to the rule that PSyclone only runs on code that has had all necessary pre-processing already performed.

[stevemullerworth]

I think the first should be OK. Currently we supply the top-level directory to the whole code tree for telling PSyclone where the kernels live, which seems to work fine.
For the 2nd, the kernels already have the use statements. It's just the modules with configuration information that are generated. The build system could deal with a two-pass auto-generation but it could get ugly. LFRic may prefer to consider a metadata solution that would allow you to have a common approach with OpenCL by promoting the use statements to the PSy-layer. We can discuss amongst ourselves and with you. What's your timescale for progressing this?

[rupertford]

This is an interesting coordination problem.

What happens at the moment (or what are the plans for a future build system)? Do they run interleaved as part of a single build phase (or are they planned to)?

I think PSyclone could promote use statements from kernels to the PSy layer without needing to have access to the contents of the module referenced in the use statement. That would mean that the modules themselves would not need to exist (or be visible) at the time

An alternative would be for the auto-generation systems to coordinate. For example, a PSyclone script could let the Configurator know to add in appropriate directives when it generated the modules?

[arporter]

use statements within kernels are a problem for both OpenCL and OpenACC and we had hoped to solve them in the same way - by turning any variables accessed in this way into kernel arguments (the only option for OpenCL). However, OpenACC also lets us insert directives into the module(s) that is/are being use'd. I think that's the solution that @rupertford is talking about and he's right - we don't then need to know any details about the variable in question. Obviously, to promote a variable to a kernel argument we'll need to add a declaration for it to the kernel and for that we do need details on its type etc.

[stevemullerworth]

@rupertford the build system works by copying existing source into a working directory, then running the code generators and putting the code into the working directory, then compiling. The "dependerater" works out dependencies: so, for example, if I modify a generated *_psy.f90 file it won't rerun PSyclone on the x90 file that produced it.

I would say you should do the appropriate solution for PSyclone, but it would be useful to know what it is as we are about to do a redesign of fcm make (the Met Office build system) taking into account code generation requirements. So far all our requirements have been a single-pass of a code generator whereas we may need to think about two-passes - configurator to create namelist reading code then PSyclone to add directives.

[rupertford]

Thanks @stevemullerworth. If you are re-designing fcm to support code generation and translation then I would humbly suggest that you build in the possibility of multiple passes (not fixing to one or two) irrespective of what we end up agreeing for PSyclone.

[TeranIvy]

Hi @stevemullerworth and @rupertford. I am also in favour of multiple passes, as well as configurator having capability to use different PSyclone command-line options and/or transformation scripts for different Fortran source files - even different invokes in the same source file if we want to do fine-grained optimisations. We currently only have one makefile for PSyclone in the LFRic build system (and one optimisation script, but this is a bit easier to change) so having different command-line options for each miniapp is impossible. We cannot really have a more detailed and varied application of PSyclone until we have this flexibility.

[sergisiso]

Right now psyclone has the -I option:

-I INCLUDE, --include INCLUDE
                        path to Fortran INCLUDE files (nemo API only)

that searches include files in the given list of directories for the Nemo API. We agreed with @arporter that we can use the same parameter to provide the list of directories where use statements search for Fortran modules (with the 'name'_mod.f90 convention) to all APIs.

Once the module is found and parsed into a PSyIR container, we need a new PSyIR Transformation to promote symbol_table globals (e.g. omega: <deferred, Scalar, global=FortranModule(physical_params_mod)>) into the psy layer, and pass them into the kernel as an argument. We need to check for namespace clashes in the psy-layer.

Complementary, some globals are defined as parameters (e.g. REAL(go_wp), PARAMETER :: pi = 3.1415926535897932_go_wp) in this cases it may be easier to just replace the Reference to the specific Literal in a prior transformation.

Applying this 2 transformation will improve the coverage (and performance?) of the generated OpenCL and OpenACC kernels.

[arporter]

#555 has been merged and #633 has been created for the job of replacing parameters used in kernels with their associated values.