Final tests fix, doc updates

* 2D restarts needed another option
* actually remove conducting_atmo from NVHPC tests
* Point people to the wiki now that it's """good"""

README.md

@@ -1,7 +1,9 @@
 # KHARMA
-KHARMA is an implementation of the HARM scheme for gerneral relativistic magnetohydrodynamics (GRMHD) in C++.  It is based on the Parthenon AMR infrastructure, using Kokkos for parallelism and GPU support.  It is composed of modular "packages," which in theory make it easy to add or swap components representing different physical processes.
+KHARMA is an implementation of the HARM scheme for gerneral relativistic magnetohydrodynamics (GRMHD) in C++.  It is based on the Parthenon AMR framework, using Kokkos for parallelism and GPU support.  It is composed of modular "packages," which in theory make it easy to add or swap components representing different algorithmic components or physics processes.
 
-KHARMA is capable of closely matching other HARM implementations, e.g. [iharm3d](https://github.com/AFD-Illinois/iharm3d). However, it also extends the scheme with additional options for magnetic field transport, reconstruction, etc.  Notably, it implements a split face-centered CT scheme, allowing static and adaptive mesh refinement.
+KHARMA is capable of closely matching other HARM implementations, e.g. [iharm3d](https://github.com/AFD-Illinois/iharm3d). However, it also updates the scheme to support static and adaptive mesh refinement, new methods for primitive variable recovery, new boundary conditions, and new stability features for running difficult simulations at high resolutions reliably.
+
+There is a bunch of documentation on the [wiki](https://github.com/AFD-Illinois/kharma/wiki).  If you have a basic question, it might be answered there!
 
 ## Prerequisites
 KHARMA requires that the system have a C++17-compliant compiler, MPI, and parallel HDF5.  All other dependencies are included as submodules, and can be checked out with `git` by running
@@ -20,23 +22,25 @@ On directly supported systems, or systems with standard install locations, you m
 ```bash
 ./make.sh clean [cuda hip sycl]
 ```
-after a *successful* compile, subsequent invocations can omit `clean`.  If this command fails on supported machines (those with a file in `machines/`), please open an issue.  Broken builds aren't uncommon, as HPC machines change software all the time.
+after a successful configuration (after you see `-- Generating done (X.Ys)`), subsequent invocations can omit `clean`.  If this command fails on supported machines (those with a file in `machines/`), please open an issue.  Broken builds aren't uncommon, as HPC machines change software all the time.
 
-If running KHARMA on a new machine (or repairing the build on an old one), take a look at the [wiki page](https://github.com/AFD-Illinois/kharma/wiki/Building-KHARMA) describing the build system.
+If (when) you run into any trouble, take a look at the [wiki page](https://github.com/AFD-Illinois/kharma/wiki/Building-KHARMA) describing the build system.
 
 ## Running
 Run a particular problem with e.g.
 ```bash
-$ ./kharma.host -i pars/tests/orszag_tang.par
+$ ./run.sh -i pars/tests/orszag_tang.par
 ```
 note that *all* options are runtime.  The single KHARMA binary can run any of the parameter files in `pars/`, and indeed this is checked as a part of the regression tests.  Note you can still disable some sub-systems manually at compile time, and of course in that case the accompanying problems will crash.
 
-KHARMA benefits from certain runtime environment variables and CPU pinning, included in a short wrapper script `run.sh`.  This script is provided mostly as an optional convenience, and an example of how to construct your own batch scripts for running KHARMA in production.  Other example batch scripts are in the `scripts/batch/` folder.
+As a broad and capable code, KHARMA has quite a lot of options.  Most are documented [here](https://github.com/AFD-Illinois/kharma/wiki/Parameters), with specific problem setups described [here](https://github.com/AFD-Illinois/kharma/wiki/Problems).
+
+Using `run.sh` is not necessary, feel free to use `kharma.host` or `kharma.cuda` directly.  The script is provided mostly to load any modules or environment variables a machine needs (again, soruced from the file in `machines/`), regardless of whether you're running interactively or as part of a batch script.
 
 Further information can be found on the [wiki page](https://github.com/AFD-Illinois/kharma/wiki/Running-KHARMA).
 
 ## Hacking
-KHARMA has some preliminary documentation for developers, hosted in its GitHub [wiki](https://github.com/AFD-Illinois/kharma/wiki).
+KHARMA has some documentation for developers on the [wiki](https://github.com/AFD-Illinois/kharma/wiki).  The docs cover some quirks of coding in C++, in particular with Kokkos/GPU programming, and in particular with Parthenon.
 
 ## Licenses
 KHARMA is made available under the BSD 3-clause license included in each file and in the file LICENSE at the root of this repository.

pars/tests/mhdmodes.par

@@ -19,8 +19,6 @@ nmode = 1
 dir = 3
 
 # Size and parameters of the full mesh
-# First two parameters are the defaults,
-# and specify no SMR/AMR
 <parthenon/mesh>
 nx1 = 64
 x1min = 0.0
@@ -62,7 +60,12 @@ gamma = 1.333333
 implicit = false
 
 <flux>
+# Local Lax-Friedrichs fluxes
+# `hlle` selects Harten, Lax, van Leer & Einfeldt
+# No production problems use HLLE as it has failure modes
+# we do not test for.
 type = llf
+# Lots of reconstruction options, see the wiki
 reconstruction = weno5
 
 <b_field>

scripts/ci/nvhpc.yml

@@ -74,5 +74,5 @@ tests:
       # This should mirror the CPU tests *except* no driven turbulence test!
       # However, NVC++ mis-compiles our EMHD kernels so we do not test the following:
       # anisotropic_conduction, bondi_viscous, conducting_atmosphere, emhdmodes,
-      - TEST: [all_pars, bondi, bz_monopole, conducting_atmosphere, mhdmodes,
-               mhdmodes_smr, noh, regrid, reinit, resize, restart, tilt_init, torus_sanity]
+      - TEST: [all_pars, bondi, bz_monopole, mhdmodes, mhdmodes_smr, noh,
+               regrid, reinit, resize, restart, tilt_init, torus_sanity]

tests/restart/run.sh

@@ -71,14 +71,15 @@ test_restart_smr() {
     fi
 }
 
-test_restart kharma "driver/type=kharma" "KHARMA driver"
-test_restart imex "driver/type=imex" "ImEx driver"
+test_restart kharma "driver/type=kharma b_field/solver=flux_ct" "KHARMA driver"
+test_restart imex "driver/type=imex b_field/solver=flux_ct" "ImEx driver"
 #test_restart imex_emhd "driver/type=imex emhd/on=true" "ImEx driver, EMHD"
 test_restart kharma_face "driver/type=kharma b_field/solver=face_ct" "KHARMA driver, face CT"
 test_restart imex_face "driver/type=imex b_field/solver=face_ct" "ImEx driver, face CT"
-TWO_D="boundaries/inner_x2=reflecting boundaries/outer_x2=reflecting parthenon/mesh/nx3=1 parthenon/meshblock/nx3=1"
-test_restart kharma_face_2d "driver/type=kharma b_field/solver=face_ct $TWO_D" "KHARMA driver, face CT, 2D"
-test_restart imex_face_2d   "driver/type=imex b_field/solver=face_ct $TWO_D" "ImEx driver, face CT, 2D"
+TWO_D="parthenon/mesh/nx3=1 parthenon/meshblock/nx3=1"
+REFLECTING="boundaries/inner_x2=reflecting boundaries/outer_x2=reflecting boundaries/excise_polar_flux=false"
+test_restart kharma_face_2d "driver/type=kharma b_field/solver=face_ct $TWO_D $REFLECTING" "KHARMA driver, face CT, 2D"
+test_restart imex_face_2d   "driver/type=imex b_field/solver=face_ct $TWO_D $REFLECTING" "ImEx driver, face CT, 2D"
 # SMR
 test_restart_smr kharma_face_smr "driver/type=kharma b_field/solver=face_ct" "KHARMA driver, face CT, SMR"
 test_restart_smr imex_face_smr "driver/type=imex b_field/solver=face_ct" "ImEx driver, face CT, SMR"