Better tool support (#309)

Tools are now represented by classes which are instantiated and placed in a "tool box." This improves the organisation and management of tools.

.gitignore

@@ -3,6 +3,11 @@ __pycache__/
 *.py[cod]
 *$py.class
 
+# Build directory for documentation
+docs/build
+docs/source/api
+docs/source/apidoc
+
 # C extensions
 *.so
 

docs/source/index.rst

@@ -80,6 +80,7 @@ See also
    config_intro
    writing_config
    advanced_config
+   site-specific-config
    features
    Api Reference <api>
    development

docs/source/site-specific-config.rst

@@ -0,0 +1,159 @@
+.. _site_specific_config:
+
+Site-Specific Configuration
+***************************
+A site might have compilers that Fab doesn't know about, or prefers
+a different compiler from the Fab default. Fab abstracts the compilers
+and other programs required during building as an instance of a
+:class:`~fab.tools.Tool` class. All tools that Fab knows about, are
+available in a :class:`~fab.tools.tool_repository.ToolRepository`.
+That will include tools that might not be available on the current system.
+
+Each tool belongs to a certain category of
+:class:`~fab.tool.category.Category`. A `ToolRepository` can store
+several instances of the same category.
+
+At build time, the user has to create an instance of
+:class:`~fab.tools.tool_box.ToolBox` and pass
+it to the :class:`~fab.build_config.BuildConfig` object. This toolbox
+contains all the tools that will be used during the build process, but
+it can only store one tool per category. If a certain tool should not
+be defined in the toolbox, the default from the `ToolRepository` will
+be used. This is useful for many standard tools like `git`, `rsync`
+etc that de-facto will never be changed.
+
+.. note:: If you need to use for example different compilers for
+          different files, you would implement this as a `meta-compiler`:
+          implement a new class based on the existing
+          :class:`~fab.tools.compiler.Compiler` class,
+          which takes two (or more) compiler instances. Its
+          :func:`~fab.tools.compiler.Compiler.compile_file`
+          method can then decide (e.g. based on the path of the file to
+          compile, or a hard-coded set of criteria) which compiler to use.
+
+Category
+==========
+All possible categories are defined in
+:class:`~fab.tool.category.Category`. If additional categories
+should be required, they can be added.
+
+Tool
+====
+Each tool must be derived from :class:`~fab.tools.Tool`.
+The base class provides a `run` method, which any tool can
+use to execute a command in a shell. Typically, a tool will
+provide one (or several) custom commands to be used by the steps.
+For example, a compiler instance provides a
+:func:`~fab.tools.compiler.Compiler.compile_file` method.
+This makes sure that no tool-specific command line options need
+to be used in any Fab step, which will allow the user to replace any tool
+with a different one.
+
+New tools can easily be created, look at
+:class:`~fab.tools.compiler.Gcc` or
+:class:`~fab.tools.compiler.Icc`. Typically, they can just be
+created by providing a different set of parameters in the
+constructor.
+
+This also allows compiler wrappers to be easily defined. For example,
+if you want to use `mpif90` as compiler, which is a MPI-specific
+wrapper for `ifort`, you can create this class as follows:
+
+.. code-block::
+    :linenos:
+    :caption: Compiler wrapper
+
+    from fab.tools import Ifort
+
+    class MpiF90(Ifort):
+        '''A simple compiler wrapper'''
+        def __init__(self):
+            super().__init__(name="mpif90-intel",
+                             exec_name="mpif90")
+
+.. note:: In `ticket 312 <https://github.com/metomi/fab/issues/312>`_ a better
+        implementation of compiler wrapper will be implemented.
+
+Tool Repository
+===============
+The :class:`~fab.tools.tool_repository.ToolRepository` implements
+a singleton to access any tool that Fab knows about. A site-specific
+startup section can add more tools to the repository:
+
+.. code-block::
+    :linenos:
+    :caption: ToolRepository
+
+    from fab.tools import ToolRepository
+
+    # Assume the MpiF90 class as shown in the previous example
+
+    tr = ToolRepository()
+    tr.add_tool(MpiF90)   # the tool repository will create the instance
+
+Compiler and linker objects define a compiler suite, and the `ToolRepository`
+provides
+:func:`~fab.tools.tool_repository.ToolRepository.set_default_compiler_suite`
+which allows you to change the defaults for compiler and linker with
+a single call. This will allow you to easily switch from one compiler
+to another. If required, you can still change any individual compiler
+after setting a default compiler suite, e.g. you can define `intel-classic`
+as default suite, but set the C-compiler to be `gcc`.
+
+
+Tool Box
+========
+The class :class:`~fab.tools.tool_box.ToolBox` is used to provide
+the tools to be used by the build environment, i.e. the
+`BuildConfig` object:
+
+.. code-block::
+    :linenos:
+    :caption: ToolBox
+
+    from fab.tools import Category, ToolBox, ToolRepository
+
+    tr = ToolRepository()
+    tr.set_default_compiler_suite("intel-classic")
+    tool_box = ToolBox()
+    ifort = tr.get_tool(Category.FORTRAN_COMPILER, "ifort")
+    tool_box.add_tool(ifort)
+    c_compiler = tr.get_default(Category.C_COMPILER)
+    tool_box.add_tool(c_compiler)
+
+    config = BuildConfig(tool_box=tool_box,
+                         project_label=f'lfric_atm-{ifort.name}', ...)
+
+The advantage of finding the compilers to use in the tool box is that
+it allows a site to replace a compiler in the tool repository (e.g.
+if a site wants to use an older gfortran version, say one which is called
+`gfortran-11`). They can then remove the standard gfortran in the tool
+repository and replace it with a new gfortran compiler that will call
+`gfortran-11` instead of `gfortran`. But a site can also decide to
+not support a generic `gfortran` call, instead adding different
+gfortran compiler with a version number in the name.
+
+If a tool category is not defined in the `ToolBox`, then
+the default tool from the `ToolRepository` will be used. Therefore,
+in the example above adding `ifort` is not strictly necessary (since
+it will be the default after setting the default compiler suite to
+`intel-classic`), and `c_compiler` is the default as well. This feature
+is especially useful for the many default tools that Fab requires (git,
+rsync, ar, ...).
+
+.. code-block::
+    :linenos:
+    :caption: ToolBox
+
+    tool_box = ToolBox()
+    default_c_compiler = tool_box.get_tool(Category.C_COMPILER)
+
+
+TODO
+====
+At this stage compiler flags are still set in the corresponding Fab
+steps, and it might make more sense to allow their modification and
+definition in the compiler objects.
+This will allow a site to define their own set of default flags to
+be used with a certain compiler by replacing or updating a compiler
+instance in the Tool Repository

pyproject.toml

@@ -48,3 +48,9 @@ version = {attr = 'fab.__version__'}
 [build-system]
 requires = ['setuptools']
 build-backend = 'setuptools.build_meta'
+
+# This is required so that pytest finds conftest.py files.
+[tool.pytest.ini_options]
+testpaths = [
+    "tests",
+]

run_configs/build_all.py

@@ -4,20 +4,48 @@
 #  For further details please refer to the file COPYRIGHT
 #  which you should have received as part of this distribution
 # ##############################################################################
+
+'''A top-level build script that executes all scripts in the various
+subdirectories.
+'''
+
 import os
 from pathlib import Path
+import shutil
+
+from fab.tools import Category, Tool, ToolBox
 
-from fab.steps.compile_fortran import get_fortran_compiler
-from fab.tools import run_command
 
+class Script(Tool):
+    '''A simple wrapper that runs a shell script.
+    :name: the path to the script to run.
+    '''
+    def __init__(self, name: Path):
+        super().__init__(name=name.name, exec_name=name,
+                         category=Category.MISC)
 
-# todo: run the exes, check the output
+    def check_available(self):
+        '''Since there typically is no command line option we could test for
+        the tolls here, we use `which` to determine if a tool is available.
+        '''
+        out = shutil.which(self.exec_name)
+        if out:
+            return True
+        print(f"Tool '{self.name}' (f{self.exec_name}) cannot be executed.")
+        return False
+
+
+# todo: after running the execs, check the output
 def build_all():
+    '''Build all example codes here.
+    '''
 
+    tool_box = ToolBox()
+    compiler = tool_box[Category.FORTRAN_COMPILER]
     configs_folder = Path(__file__).parent
-    compiler, _ = get_fortran_compiler()
 
-    os.environ['FAB_WORKSPACE'] = os.path.join(os.getcwd(), f'fab_build_all_{compiler}')
+    os.environ['FAB_WORKSPACE'] = \
+        os.path.join(os.getcwd(), f'fab_build_all_{compiler.name}')
 
     scripts = [
         configs_folder / 'tiny_fortran/build_tiny_fortran.py',
@@ -38,20 +66,22 @@ def build_all():
 
     # skip these for now, until we configure them to build again
     compiler_skip = {'gfortran': [], 'ifort': ['atm.py']}
-    skip = compiler_skip[compiler]
+    skip = compiler_skip[compiler.name]
 
     for script in scripts:
-
+        script_tool = Script(script)
         # skip this build script for the current compiler?
         if script.name in skip:
             print(f''
                   f'-----'
-                  f'SKIPPING {script.name} FOR COMPILER {compiler} - GET THIS COMPILING AGAIN'
+                  f'SKIPPING {script.name} FOR COMPILER {compiler.name} - '
+                  f'GET THIS COMPILING AGAIN'
                   f'-----')
             continue
 
-        run_command([script], capture_output=False)
+        script_tool.run(capture_output=False)
 
 
+# =============================================================================
 if __name__ == '__main__':
     build_all()

run_configs/gcom/build_gcom_ar.py

@@ -4,15 +4,18 @@
 # For further details please refer to the file COPYRIGHT
 # which you should have received as part of this distribution
 ##############################################################################
+
 from fab.build_config import BuildConfig
 from fab.steps.archive_objects import archive_objects
 from fab.steps.cleanup_prebuilds import cleanup_prebuilds
+from fab.tools import ToolBox
 from gcom_build_steps import common_build_steps
 
 
 if __name__ == '__main__':
 
-    with BuildConfig(project_label='gcom object archive $compiler') as state:
+    with BuildConfig(project_label='gcom object archive $compiler',
+                     tool_box=ToolBox()) as state:
         common_build_steps(state)
         archive_objects(state, output_fpath='$output/libgcom.a')
         cleanup_prebuilds(state, all_unused=True)

run_configs/gcom/build_gcom_so.py

@@ -4,6 +4,8 @@
 # For further details please refer to the file COPYRIGHT
 # which you should have received as part of this distribution
 ##############################################################################
+
+from fab.tools import ToolBox
 from fab.build_config import BuildConfig
 from fab.steps.cleanup_prebuilds import cleanup_prebuilds
 from fab.steps.link import link_shared_object
@@ -17,7 +19,8 @@
     # we can add our own arguments here
     parsed_args = arg_parser.parse_args()
 
-    with BuildConfig(project_label='gcom shared library $compiler') as state:
+    with BuildConfig(project_label='gcom shared library $compiler',
+                     tool_box=ToolBox()) as state:
         common_build_steps(state, fpic=True)
         link_shared_object(state, output_fpath='$output/libgcom.so'),
         cleanup_prebuilds(state, all_unused=True)

run_configs/gcom/grab_gcom.py

@@ -4,14 +4,17 @@
 # For further details please refer to the file COPYRIGHT
 # which you should have received as part of this distribution
 ##############################################################################
+
 from fab.build_config import BuildConfig
 from fab.steps.grab.fcm import fcm_export
+from fab.tools import ToolBox
 
 
 revision = 'vn7.6'
 
 # we put this here so the two build configs can read its source_root
-grab_config = BuildConfig(project_label=f'gcom_source {revision}')
+grab_config = BuildConfig(project_label=f'gcom_source {revision}',
+                          tool_box=ToolBox())
 
 
 if __name__ == '__main__':

run_configs/jules/build_jules.py

@@ -17,20 +17,36 @@
 from fab.steps.link import link_exe
 from fab.steps.preprocess import preprocess_fortran
 from fab.steps.root_inc_files import root_inc_files
+from fab.tools import Ifort, Linker, ToolBox
 
 logger = logging.getLogger('fab')
 
 
+# TODO 312: we need to support non-intel compiler here.
+
+class MpiIfort(Ifort):
+    '''A small wrapper to make mpif90 available.'''
+    def __init__(self):
+        super().__init__(name="mpif90", exec_name="mpif90")
+
+
 if __name__ == '__main__':
 
     revision = 'vn6.3'
 
-    with BuildConfig(project_label=f'jules {revision} $compiler') as state:
+    tool_box = ToolBox()
+    # Create a new Fortran compiler MpiIfort
+    fc = MpiIfort()
+    tool_box.add_tool(fc)
+    # Use the compiler as linker:
+    tool_box.add_tool(Linker(compiler=fc))
+
+    with BuildConfig(project_label=f'jules {revision} $compiler',
+                     tool_box=tool_box) as state:
         # grab the source. todo: use some checkouts instead of exports in these configs.
         fcm_export(state, src='fcm:jules.xm_tr/src', revision=revision, dst_label='src')
         fcm_export(state, src='fcm:jules.xm_tr/utils', revision=revision, dst_label='utils')
 
-        #
         grab_pre_build(state, path='/not/a/real/folder', allow_fail=True),
 
         # find the source files
@@ -47,12 +63,12 @@
 
         preprocess_fortran(state, common_flags=['-P', '-DMPI_DUMMY', '-DNCDF_DUMMY', '-I$output'])
 
-        analyse(state, root_symbol='jules', unreferenced_deps=['imogen_update_carb']),
+        analyse(state, root_symbol='jules', unreferenced_deps=['imogen_update_carb'])
 
         compile_fortran(state)
 
-        archive_objects(state),
+        archive_objects(state)
 
-        link_exe(state, linker='mpifort', flags=['-lm', '-lnetcdff', '-lnetcdf']),
+        link_exe(state, flags=['-lm', '-lnetcdff', '-lnetcdf'])
 
         cleanup_prebuilds(state, n_versions=1)