Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NSF NCAR example #36

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add NSF NCAR example #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fe5cbd4
Add NSF NCAR example
dphow Feb 25, 2025
ff9247f
Add default /glade/u/home/dhoward/.julia path for backwards copatibil…
dphow Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions examples/ncar/derecho/ncar_derecho_config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash

# Author: Daniel Howard, NSF NCAR (dhoward)

# Load required modules (including correct CPU and GPU target modules)
ml ncarenv/24.12
ml reset
ml gcc/12.4.0
ml cuda/12.3.2
ml swap hdf5 hdf5-mpi/1.12.3
module list

# Environment variables for HPC key packages that require system libraries that require system libraries (MPI.jl, CUDA.jl, AMDGPU.jl, HDF5.jl and ADIOS2.jl)
export JUHPC_CUDA_HOME=$CUDA_HOME
export JUHPC_CUDA_RUNTIME_VERSION=$CRAY_CUDATOOLKIT_VERSION
export JUHPC_MPI_VENDOR="cray"
export JUHPC_MPI_EXEC="mpiexec"
export JUHPC_HDF5_HOME=$NCAR_ROOT_HDF5

# Call JUHPC
JUHPC_SETUP_INSTALLDIR=/glade/u/apps/opt/julia/$NCAR_HOST/juhpc_setup
JULIAUP_INSTALLDIR="/glade/work/\$USER/julia/\$NCAR_HOST/juliaup"
VERSION="v0.3.0"
#wget https://raw.githubusercontent.com/JuliaParallel/JUHPC/$VERSION/juhpc -O ./juhpc
./juhpc $JUHPC_SETUP_INSTALLDIR $JULIAUP_INSTALLDIR --verbose=2 | tee juhpc_install.log
Copy link
Collaborator

@omlins omlins Mar 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It automatically creates a log file as described here: .https://github.com/JuliaParallel/JUHPC?tab=readme-ov-file#2-call-juhpc
(code: https://github.com/JuliaParallel/JUHPC/blob/main/juhpc#L73)

33 changes: 33 additions & 0 deletions examples/ncar/derecho/test_ncar_derecho_config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash

# Load required modules (including correct CPU and GPU target modules)
ml ncarenv/24.12
ml reset
ml gcc/12.4.0
ml cuda/12.3.2
ml swap hdf5 hdf5-mpi/1.12.3
module list

# Call JUHPC
JUHPC_SETUP_INSTALLDIR=/glade/u/apps/opt/julia/$NCAR_HOST/juhpc_setup

# Activate the HPC setup environment variables
. $JUHPC_SETUP_INSTALLDIR/activate

# Call juliaup to install juliaup and latest julia on scratch
juliaup

# Call juliaup to see its options
juliaup

# Call julia Pkg
julia -e 'using Pkg; Pkg.status()'

# Add CUDA.jl
julia -e 'using Pkg; Pkg.add("CUDA"); using CUDA; CUDA.versioninfo()'

# Add MPI.jl
julia -e 'using Pkg; Pkg.add("MPI"); using MPI; MPI.versioninfo()'

# Add HDF5.jl
julia -e 'using Pkg; Pkg.add("HDF5"); using HDF5; @show HDF5.has_parallel()'
9 changes: 6 additions & 3 deletions juhpc
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,9 @@ fi

export JULIAUP_BINDIR="$JULIAUP_INSTALLDIR/bin" # juliaup and julia binaries

if [ -d "/dev/shm" ]; then
if [ ! -z "$NCAR_HOST"]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in order to support your use case, I think it would be better to add a keyword argument --tmp and create internally in the variable JULIAUP_TMP (here:

JUHPC/juhpc

Lines 126 to 139 in ff9247f

# Parse and check the keyword arguments
shift 2
for arg in "$@"; do
case $arg in
--postinstall=*) export JUHPC_POST_INSTALL_JL="${arg#*=}" ;;
--verbose=*) export JUHPC_VERBOSE="${arg#*=}" ;;
*) error "Unknown argument: $arg" ;;
esac
done
if ! [[ "$JUHPC_VERBOSE" =~ ^[0-2]$ ]]; then
error "JUHPC_VERBOSE must be between 0 and 2."
fi
). Then, it can be assigned to TMP if it is set and else do as before.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only other change was setting TMP directory to NCAR scratch (our scratch purges) though the solution might be applied differently for general future users.

By the way: why do you want to do that? Do you not have /tmp or /dev/shm on your system? Note that this is only used for a temporary installation of julia required to do the setup; the installation will be removed in the end.

export TMP="$SCRATCH/tmp"
elif [ -d "/dev/shm" ]; then
export TMP="/dev/shm/$USER"
elif [ -d "/tmp" ]; then
export TMP="/tmp/$USER"
Expand Down Expand Up @@ -301,7 +303,8 @@ info "... done: wrapper created."
info "Creating activate script..."

export JULIAUP_DEPOT="$JULIAUP_INSTALLDIR/depot"
export JULIA_DEPOT="$JULIAUP_INSTALLDIR/depot"
# Added default $HOME/.julia path for backwards compatibility with exisitng Julia user installs
export JULIA_DEPOT="$JULIAUP_INSTALLDIR/depot:$HOME/.julia"
Copy link
Collaborator

@omlins omlins Mar 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that it is a good idea to mix with user installations. Also, we can have many different JUHPC setups that don't interfere with each other; adding this common depot or would break this great feature.

export ACTIVATE_SCRIPT="$JUHPC_SETUP_INSTALLDIR/activate"

julia -e 'println("""
Expand Down Expand Up @@ -356,4 +359,4 @@ cleanup
mv "$JULIA_PREFDIR/Manifest.toml" "$JULIA_PREFDIR/Manifest.toml.bak" || { error "failed to move Manifest.toml in $JULIA_PREFDIR"; }
mv "$JUHPC_SETUP_INSTALLLOG" "$JUHPC_SETUP_INSTALLDIR/" || { error "failed to move $JUHPC_SETUP_INSTALLLOG to $JUHPC_SETUP_INSTALLDIR"; }

info "... The installation of the HPC setup for juliaup, julia and HPC key packages is complete.\n\n"
info "... The installation of the HPC setup for juliaup, julia and HPC key packages is complete.\n\n"