Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First-time user observations and error #28

Open
paulmelis opened this issue Oct 15, 2024 · 18 comments · Fixed by #29
Open

First-time user observations and error #28

paulmelis opened this issue Oct 15, 2024 · 18 comments · Fixed by #29

Comments

@paulmelis
Copy link

paulmelis commented Oct 15, 2024

First up, great that someone's working on making Julia use on HPC systems easier!

As I'm now trying JUHPC for the first time here's some things I noticed/ran into. Background: viz guy at Dutch HPC site, so consider me an "internal user" of our system :)

First, there's a lot of output coming not just from JUHPC, but also some of the programs it calls itself. This is a bit confusing, as it's not immediately clear what to pay attention to and what can be ignored. Example below, apparently stuff gets installed in a tempdir, which resolves to /dev/shm on our system, which is fairly non-standard, but okay. The thing I wasn't clear on was if "You can uninstall at any time with juliaup self uninstall and these changes will be reverted" was something I needed to remember or not. It's only at the end when it reads "cleaning up temporary juliaup installation in /dev/shm/paulm/juliaup" that I figured all the /dev/shm stuff was not relevant for the end result. Perhaps putting all that output in a log file would be better, as it hides unnecessary details that aren't of interest to the average user.

...
info: downloading installer
Welcome to Julia!

This will download and install the official Julia Language distribution
and its version manager Juliaup.

Juliaup will be installed into the Juliaup home directory, located at:

  /dev/shm/paulm/juliaup

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /dev/shm/paulm/juliaup/bin

You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.
...
[ JUHPC: cleaning up temporary juliaup installation in /dev/shm/paulm/juliaup.
[ JUHPC: ... The installation of the HPC setup for juliaup, julia and HPC key packages is complete.

For the lines below a similar thing applies, as I assume the "re-start Julia" instruction really is just a detail that isn't relevant for JUHPC installation.

Precompiling project...
  75 dependencies successfully precompiled in 322 seconds. 23 already precompiled.
[ Info: Configure the active project to use CUDA 12.6 from the local system; please re-start Julia for this to take effect.

After then trying to activate the installed JUHPC environment it seems the first steps are okay:

snellius paulm@tcn965 14:30 ~/c/JUHPC$ module list

Currently Loaded Modules:
  1) 2024                           5) GCC/13.3.0                      9) libpciaccess/0.18.1-GCCcore-13.3.0  13) UCX/1.16.0-GCCcore-13.3.0        17) UCC/1.3.0-GCCcore-13.3.0
  2) GCCcore/13.3.0                 6) numactl/2.0.18-GCCcore-13.3.0  10) hwloc/2.10.0-GCCcore-13.3.0         14) libfabric/1.21.0-GCCcore-13.3.0  18) OpenMPI/5.0.3-GCC-13.3.0
  3) zlib/1.3.1-GCCcore-13.3.0      7) XZ/5.4.5-GCCcore-13.3.0        11) OpenSSL/3                           15) PMIx/5.0.2-GCCcore-13.3.0        19) gompi/2024a
  4) binutils/2.42-GCCcore-13.3.0   8) libxml2/2.12.7-GCCcore-13.3.0  12) libevent/2.1.12-GCCcore-13.3.0      16) PRRTE/3.0.5-GCCcore-13.3.0       20) CUDA/12.6.0

snellius paulm@tcn965 14:30 ~/c/JUHPC$ source /scratch-shared/paulm/juhpc_setup_genoa/activate 
[ JUHPC: Activating HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: PATH: /scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juhpc_setup_genoa/bin:/sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0/nvvm/bin:/sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0/bin:/sw/arch/RHEL9/EB_production/2024/software/OpenMPI/5.0.3-GCC-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/UCC/1.3.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/PRRTE/3.0.5-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/PMIx/5.0.2-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libfabric/1.21.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/UCX/1.16.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libevent/2.1.12-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/OpenSSL/3/bin:/sw/arch/RHEL9/EB_production/2024/software/hwloc/2.10.0-GCCcore-13.3.0/sbin:/sw/arch/RHEL9/EB_production/2024/software/hwloc/2.10.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libxml2/2.12.7-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/XZ/5.4.5-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/numactl/2.0.18-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/binutils/2.42-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/GCCcore/13.3.0/bin:/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juhpc_setup_genoa/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/home/paulm/bin:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[ JUHPC: JULIAUP_DEPOT_PATH: /scratch-shared/paulm/juhpc_setup_genoa/depot
[ JUHPC: JULIA_DEPOT_PATH: /scratch-shared/paulm/juhpc_setup_genoa/depot
[ JUHPC: JULIA_LOAD_PATH: :/scratch-shared/paulm/juhpc_setup_genoa/julia_preferences
[ JUHPC: CUDA_HOME: /sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0
[ JUHPC: JULIA_CUDA_MEMORY_POOL: none

But not having a julia available is somewhat surprising:

snellius paulm@tcn965 14:31 ~/c/JUHPC$ which julia
/usr/bin/which: no julia in (/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juhpc_setup_genoa/bin:/sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0/nvvm/bin:/sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0/bin:/sw/arch/RHEL9/EB_production/2024/software/OpenMPI/5.0.3-GCC-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/UCC/1.3.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/PRRTE/3.0.5-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/PMIx/5.0.2-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libfabric/1.21.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/UCX/1.16.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libevent/2.1.12-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/OpenSSL/3/bin:/sw/arch/RHEL9/EB_production/2024/software/hwloc/2.10.0-GCCcore-13.3.0/sbin:/sw/arch/RHEL9/EB_production/2024/software/hwloc/2.10.0-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/libxml2/2.12.7-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/XZ/5.4.5-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/numactl/2.0.18-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/binutils/2.42-GCCcore-13.3.0/bin:/sw/arch/RHEL9/EB_production/2024/software/GCCcore/13.3.0/bin:/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juhpc_setup_genoa/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/home/paulm/bin:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)

And juliaup is where I would expect it, but running that gives me an error:

snellius paulm@tcn965 14:31 ~/c/JUHPC$ which juliaup
/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper/juliaup

snellius paulm@tcn965 14:31 ~/c/JUHPC$ juliaup


     ???  ???   ???  ???  ??????????  ???????
     ???  ???   ???  ???  ???????????????????
     ???  ???   ???  ???????????????????     
???  ???  ???   ???  ??????????????? ???     
????????  ?????????  ???  ??????     ????????
 ??????    ???????   ???  ??????      ???????


[ JUHPC: This is the first call to juliaup: installing juliaup and the latest julia...

info: downloading installer
Welcome to Julia!

This will download and install the official Julia Language distribution
and its version manager Juliaup.

Juliaup will be installed into the Juliaup home directory, located at:

  /scratch-shared/paulm/juhpc_setup_genoa

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /scratch-shared/paulm/juhpc_setup_genoa/bin

You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.

You are trying to install Juliaup into the folder
`/scratch-shared/paulm/juhpc_setup_genoa`,
but that folder already exists. Please remove that folder
and then start the setup process again.
[ JUHPC: ERROR: juliaup installation failed.

The error doesn't make much sense to me, as I assumed juliaup was already installed earlier? Since the installation took quite a few minutes and processed a lot of Julia packages. So perhaps something is going wrong on my end?

So with respect to actually using (as an end-user) the installed JUHPC environment I'm unsure what to expect. For example, one of the example scripts in the repo here shows the line . $JUHPC_SETUP_INSTALLDIR/activate, but I assume that I don't need to keep $JUHPC_SETUP_INSTALLDIR set, as that would defeat the purpose of having separate install directories?

@omlins
Copy link
Collaborator

omlins commented Oct 15, 2024

Thanks for sharing this! One could indeed write a lot of information into log file instead of showing it.
All the output that is from JUHPC start with [ JUHPC:

You are trying to install Juliaup into the folder
/scratch-shared/paulm/juhpc_setup_genoa,
but that folder already exists. Please remove that folder
and then start the setup process again.

It looks like you gave the same folder for the setup installation and the juliaup installation. If you change that then it should work.

@omlins
Copy link
Collaborator

omlins commented Oct 15, 2024

You can try with VERSION="refs/heads/initial" which will use the file from the pull request. There it will write the output into a log file instead. The output looks then as follows:

[todi][omlins@todi-ln002 ~]$ bash -l /tmp/juhpc $JUHPC_SETUP_INSTALLDIR $JULIAUP_INSTALLDIR


     ██╗  ██╗   ██╗  ██╗  ██╗██████╗  ██████╗
     ██║  ██║   ██║  ██║  ██║██╔══██╗██╔════╝
     ██║  ██║   ██║  ███████║██████╔╝██║     
██╗  ██║  ██║   ██║  ██╔══██║██╔═══╝ ██║     
╚█████╔╝  ╚██████╔╝  ██║  ██║██║     ╚██████╗
 ╚════╝    ╚═════╝   ╚═╝  ╚═╝╚═╝      ╚═════╝


[ JUHPC: Starting installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: Installing juliaup and julia temporarily in /dev/shm/omlins/juliaup...

info: downloading installer
Installing Julia 1.11.0+0.aarch64.linux.gnu
Configured the default Julia version to be 'release'.
[ JUHPC: ... done: temporary installation completed.
[ JUHPC: Creating preferences for HPC key packages...
[ JUHPC: ... done: preferences created.
[ JUHPC: Creating wrapper for juliaup...
[ JUHPC: ... done.
[ JUHPC: Creating activate script...
[ JUHPC: ... done.
[ JUHPC: cleaning up temporary juliaup installation in /dev/shm/omlins/juliaup.
[ JUHPC: ... The installation of the HPC setup for juliaup, julia and HPC key packages is complete.

To avoid that people get worried because of the time nothing is happening a progress bar should be added...

@omlins
Copy link
Collaborator

omlins commented Oct 15, 2024

Note that at this point only the setup is installed. Juliaup and julia will be installed at first call of juliaup. The user who installs the set up does not have to be the same user asked the one who uses it finally... It looks then as follows when using it as in the examples:

omlins@Omlin-Lenovo:~/tmpwdir/juliadev/JUHPC$ ssh todi.cscs.ch

=========================================================================
             IMPORTANT NOTICE FOR USERS of Alps (TODI)

  Documentation: Alps User Guide - https://confluence.cscs.ch/x/_gD0E
  Request support: Service Desk at https://support.cscs.ch
=========================================================================

[todi][omlins@todi-ln001 ~]$ # Variable set in craype_config
[todi][omlins@todi-ln001 ~]$ JUHPC_SETUP_INSTALLDIR=$SCRATCH/${HOSTNAME%%-*}/juhpc_setup  # HPC setup installation environment variables must be expanded during installation.
[todi][omlins@todi-ln001 ~]$ 
[todi][omlins@todi-ln001 ~]$ # Load required modules (including correct CPU and GPU target modules)
[todi][omlins@todi-ln001 ~]$ module load cray

[todi][omlins@todi-ln001 ~]$ module switch PrgEnv-cray PrgEnv-gnu

Lmod is automatically replacing "cce/17.0.0" with "gcc-native/12.3".


Due to MODULEPATH changes, the following have been reloaded:
  1) cray-libsci/23.12.5     2) cray-mpich/8.1.28

[todi][omlins@todi-ln001 ~]$ module load cudatoolkit craype-accel-nvidia90
[todi][omlins@todi-ln001 ~]$ module load cray-hdf5-parallel
[todi][omlins@todi-ln001 ~]$ module list

Currently Loaded Modules:
  1) craype-arm-grace     4) xpmem/2.9.6-1.1_20240511212657__g087dc11fc19d   7) cray/23.12       10) cray-mpich/8.1.28    13) cudatoolkit/24.3_12.3
  2) libfabric/1.15.2.0   5) perftools-base/23.12.0                          8) gcc-native/12.3  11) cray-libsci/23.12.5  14) craype-accel-nvidia90
  3) craype-network-ofi   6) cpe/23.12                                       9) craype/2.7.30    12) PrgEnv-gnu/8.5.0     15) cray-hdf5-parallel/1.12.2.9

 

[todi][omlins@todi-ln001 ~]$ 
[todi][omlins@todi-ln001 ~]$ # Activate the HPC setup environment variables
[todi][omlins@todi-ln001 ~]$ . $JUHPC_SETUP_INSTALLDIR/activate
[ JUHPC: Activating HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: PATH: /capstor/scratch/cscs/omlins/todi/juhpc_setup/juliaup_wrapper:/capstor/scratch/cscs/omlins/todi/juliaup/bin:/opt/cray/pe/hdf5-parallel/1.12.2.9/bin:/opt/cray/pe/hdf5/1.12.2.9/bin:/opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/compute-sanitizer:/opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/bin:/opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/libnvvp:/opt/nvidia/hpc_sdk/Linux_aarch64/24.3/profilers/Nsight_Compute:/opt/nvidia/hpc_sdk/Linux_aarch64/24.3/profilers/Nsight_Systems/bin:/opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3/bin:/opt/cray/pe/mpich/8.1.28/bin:/opt/cray/pe/craype/2.7.30/bin:/opt/cray/pe/perftools/23.12.0/bin:/opt/cray/pe/papi/7.0.1.2/bin:/opt/cray/libfabric/1.15.2.0/bin:/capstor/scratch/cscs/omlins/julia_local/julia_latest/bin:/users/omlins/stackinator/bin:/usr/local/bin:/usr/bin:/bin:/usr/lpp/mmfs/bin:/usr/lib/mit/bin:/opt/cray/pe/bin
[ JUHPC: JULIAUP_DEPOT_PATH: /capstor/scratch/cscs/omlins/todi/juliaup/depot
[ JUHPC: JULIA_DEPOT_PATH: /capstor/scratch/cscs/omlins/todi/juliaup/depot
[ JUHPC: JULIA_LOAD_PATH: :/capstor/scratch/cscs/omlins/todi/juhpc_setup/julia_preferences
[ JUHPC: CUDA_HOME: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3
[ JUHPC: JULIA_CUDA_MEMORY_POOL: none
[todi][omlins@todi-ln001 ~]$ 
[todi][omlins@todi-ln001 ~]$ # Call juliaup to install juliaup and latest julia on scratch
[todi][omlins@todi-ln001 ~]$ juliaup


     ██╗  ██╗   ██╗  ██╗  ██╗██████╗  ██████╗
     ██║  ██║   ██║  ██║  ██║██╔══██╗██╔════╝
     ██║  ██║   ██║  ███████║██████╔╝██║     
██╗  ██║  ██║   ██║  ██╔══██║██╔═══╝ ██║     
╚█████╔╝  ╚██████╔╝  ██║  ██║██║     ╚██████╗
 ╚════╝    ╚═════╝   ╚═╝  ╚═╝╚═╝      ╚═════╝


[ JUHPC: This is the first call to juliaup: installing juliaup and the latest julia...

info: downloading installer
Welcome to Julia!

This will download and install the official Julia Language distribution
and its version manager Juliaup.

Juliaup will be installed into the Juliaup home directory, located at:

  /capstor/scratch/cscs/omlins/todi/juliaup

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /capstor/scratch/cscs/omlins/todi/juliaup/bin

You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.

Now installing Juliaup
Installing Julia 1.11.0+0.aarch64.linux.gnu
Configured the default Julia version to be 'release'.
Julia was successfully installed on your system.
[ JUHPC: ... done: installation completed (location: /capstor/scratch/cscs/omlins/todi/juliaup). You can now use juliaup and julia normally.

@paulmelis
Copy link
Author

You can try with VERSION="refs/heads/initial" which will use the file from the pull request. There it will write the output into a log file instead.

Okay, that indeed is a lot cleaner! Suggestion to move the log file to $JUHPC_SETUP_INSTALLDIR after juhpc is done, to have something to reference later, or in case of bugs.

It looks like you gave the same folder for the setup installation and the juliaup installation. If you change that then it should work.

Ah, indeed I messed that up. I also wasn't noticing that installation to $JUHPC_SETUP_INSTALLDIR is more or less a one-time global action, while things going to $JULIAUP_INSTALLDIR are assumed to be per-user? Because if those two dirs are related (and meant to be cleanly used together only), then why aren't they part of the same parent directory? But maybe I'm misunderstanding that part, i.e. if juhpc can be called multiple times with the same $JUHPC_SETUP_INSTALLDIR while varying $JULIAUP_INSTALLDIR.

@omlins
Copy link
Collaborator

omlins commented Oct 17, 2024

@paulmelis : I think i have addressed all the issues you raised. Could you please try again using the branch from the PR setting version as before? I also added a key word argument for verbosity. All the changes are documented here: https://github.com/JuliaParallel/JUHPC/tree/initial?tab=readme-ov-file#2-call-juhpc

@paulmelis
Copy link
Author

Yep, seems all okay now!

snellius paulm@tcn1174 19:51 ~/c/JUHPC$ m list 
No modules loaded
snellius paulm@tcn1174 19:51 ~/c/JUHPC$ cat snellius-genoa.sh
#!/bin/sh
source ./snellius-genoa.modules

export JUHPC_CUDA_HOME=$EBROOTCUDA
export JUHPC_CUDA_RUNTIME_VERSION=12.6
export JUHPC_MPI_HOME=$EBROOTOPENMPI
export JUHPC_MPI_EXEC=srun
#export HDF5

export JUHPC_SETUP_INSTALLDIR=/scratch-shared/paulm/juhpc_setup_genoa
# XXX need to escape any $ in this value
export JULIAUP_INSTALLDIR=/scratch-shared/paulm/juliaup_genoa

bash -l ./juhpc $JUHPC_SETUP_INSTALLDIR $JULIAUP_INSTALLDIR

snellius paulm@tcn1174 19:51 ~/c/JUHPC$ cat snellius-genoa.modules 
module load 2024
module load gompi/2024a
module load CUDA/12.6.0

snellius paulm@tcn1174 19:51 ~/c/JUHPC$ ./snellius-genoa.sh


     ██╗  ██╗   ██╗  ██╗  ██╗██████╗  ██████╗
     ██║  ██║   ██║  ██║  ██║██╔══██╗██╔════╝
     ██║  ██║   ██║  ███████║██████╔╝██║     
██╗  ██║  ██║   ██║  ██╔══██║██╔═══╝ ██║     
╚█████╔╝  ╚██████╔╝  ██║  ██║██║     ╚██████╗
 ╚════╝    ╚═════╝   ╚═╝  ╚═╝╚═╝      ╚═════╝


[ JUHPC: Starting installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: Installing juliaup and julia temporarily in /dev/shm/paulm/juliaup...

info: downloading installer
Installing Julia 1.11.1+0.x64.linux.gnu
Configured the default Julia version to be 'release'.
[ JUHPC: ... done: temporary installation completed.
[ JUHPC: Creating preferences for HPC key packages...
|██████████████████████████████████████████████████| 100.00%[ JUHPC: ... done: preferences created.
[ JUHPC: Creating wrapper for juliaup...
[ JUHPC: ... done: wrapper created.
[ JUHPC: Creating activate script...
[ JUHPC: ... done: activate script created.
[ JUHPC: cleaning up temporary juliaup installation in /dev/shm/paulm/juliaup.
[ JUHPC: ... The installation of the HPC setup for juliaup, julia and HPC key packages is complete.

snellius paulm@tcn1174 19:55 ~/c/JUHPC$ source /scratch-shared/paulm/juhpc_setup_genoa/activate 
[ JUHPC: Activating HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: PATH: /scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juliaup_genoa/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/home/paulm/.local/bin:/home/paulm/bin:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[ JUHPC: JULIAUP_DEPOT_PATH: /scratch-shared/paulm/juliaup_genoa/depot
[ JUHPC: JULIA_DEPOT_PATH: /scratch-shared/paulm/juliaup_genoa/depot
[ JUHPC: JULIA_LOAD_PATH: :/scratch-shared/paulm/juhpc_setup_genoa/julia_preferences
[ JUHPC: CUDA_HOME: /sw/arch/RHEL9/EB_production/2024/software/CUDA/12.6.0
[ JUHPC: JULIA_CUDA_MEMORY_POOL: none

snellius paulm@tcn1174 19:55 ~/c/JUHPC$ which julia
/usr/bin/which: no julia in (/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper:/scratch-shared/paulm/juliaup_genoa/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/gpfs/admin/hpc/usertools:/bin:/home/paulm/.local/bin:/home/paulm/software/bob:/home/paulm/.local/bin:/home/paulm/bin:/gpfs/admin/hpc/sw/hpc/bin:/gpfs/admin/hpc/sw/hpc/sbin:/opt/neovim/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
snellius paulm@tcn1174 19:55 ~/c/JUHPC$ which juliaup
/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper/juliaup
snellius paulm@tcn1174 19:55 ~/c/JUHPC$ juliaup


     ██╗  ██╗   ██╗  ██╗  ██╗██████╗  ██████╗
     ██║  ██║   ██║  ██║  ██║██╔══██╗██╔════╝
     ██║  ██║   ██║  ███████║██████╔╝██║     
██╗  ██║  ██║   ██║  ██╔══██║██╔═══╝ ██║     
╚█████╔╝  ╚██████╔╝  ██║  ██║██║     ╚██████╗
 ╚════╝    ╚═════╝   ╚═╝  ╚═╝╚═╝      ╚═════╝


[ JUHPC: This is the first call to juliaup: installing juliaup and the latest julia...

info: downloading installer
Welcome to Julia!

This will download and install the official Julia Language distribution
and its version manager Juliaup.

Juliaup will be installed into the Juliaup home directory, located at:

  /scratch-shared/paulm/juliaup_genoa

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /scratch-shared/paulm/juliaup_genoa/bin

You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.

Now installing Juliaup
Installing Julia 1.11.1+0.x64.linux.gnu
Configured the default Julia version to be 'release'.
Julia was successfully installed on your system.
[ JUHPC: ... done: installation completed (location: /scratch-shared/paulm/juliaup_genoa). You can now use juliaup and julia normally.
snellius paulm@tcn1174 19:56 ~/c/JUHPC$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.1 (2024-10-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 
snellius paulm@tcn1174 19:56 ~/c/JUHPC$ which julia
/scratch-shared/paulm/juliaup_genoa/bin/julia
snellius paulm@tcn1174 19:56 ~/c/JUHPC$ which juliaup
/scratch-shared/paulm/juhpc_setup_genoa/juliaup_wrapper/juliaup
snellius paulm@tcn1174 19:56 ~/c/JUHPC$ juliaup status
 Default  Channel  Version                 Update 
--------------------------------------------------
       *  release  1.11.1+0.x64.linux.gnu         

@omlins
Copy link
Collaborator

omlins commented Oct 18, 2024

Great! Could you do a PR to include your configuration and testing in the examples:
https://github.com/JuliaParallel/JUHPC/tree/main/examples
?

@paulmelis
Copy link
Author

paulmelis commented Oct 18, 2024

Sure, I can look into that. One thing I'm still not clear on is this point:

I also wasn't noticing that installation to $JUHPC_SETUP_INSTALLDIR is more or less a one-time global action, while things going to $JULIAUP_INSTALLDIR are assumed to be per-user? Because if those two dirs are related (and meant to be cleanly used together only), then why aren't they part of the same parent directory? But maybe I'm misunderstanding that part, i.e. if juhpc can be called multiple times with the same $JUHPC_SETUP_INSTALLDIR while varying $JULIAUP_INSTALLDIR?

@omlins
Copy link
Collaborator

omlins commented Oct 18, 2024

JUHPC_SETUP_INSTALLDIR is meant conceptually for all users or for a group of users (of course in practice it can also be for a single end user)
JULIAUP_INSTALLDIR is is always meant per single user

@omlins
Copy link
Collaborator

omlins commented Oct 18, 2024

juhpc can be called multiple times with the same $JUHPC_SETUP_INSTALLDIR while varying $JULIAUP_INSTALLDIR

No it will complain if any of the directories exist already

@omlins
Copy link
Collaborator

omlins commented Oct 18, 2024

Sure, I can look into that.

Great!

@paulmelis
Copy link
Author

paulmelis commented Oct 18, 2024

These statements seem to conflict?

JUHPC_SETUP_INSTALLDIR is meant conceptually for all users or for a group of users (of course in practice it can also be for a single end user)
JULIAUP_INSTALLDIR is is always meant per single user

versus

juhpc can be called multiple times with the same $JUHPC_SETUP_INSTALLDIR while varying $JULIAUP_INSTALLDIR

No it will complain if any of the directories exist already

Can or can't JUHPC_SETUP_INSTALLDIR be used multiple times for different JULIAUP_INSTALLDIR's?

@omlins
Copy link
Collaborator

omlins commented Oct 23, 2024

These statements seem to conflict?

JUHPC_SETUP_INSTALLDIR is meant conceptually for all users or for a group of users (of course in practice it can also be for a single end user)
JULIAUP_INSTALLDIR is is always meant per single user

versus

juhpc can be called multiple times with the same $JUHPC_SETUP_INSTALLDIR while varying $JULIAUP_INSTALLDIR

No it will complain if any of the directories exist already

Can or can't JUHPC_SETUP_INSTALLDIR be used multiple times for different JULIAUP_INSTALLDIR's?

No, the statements don't conflict. Only one HPC setup can be installed into a JUHPC_SETUP_INSTALLDIR, however, multiple users can use it and it should contain a JULIAUP_INSTALLDIR that expands differently for each user as explained here: https://github.com/JuliaParallel/JUHPC?tab=readme-ov-file#2-call-juhpc

@paulmelis
Copy link
Author

No, the statements don't conflict. Only one HPC setup can be installed into a JUHPC_SETUP_INSTALLDIR, however, multiple users can use it and it should contain a JULIAUP_INSTALLDIR that expands differently for each user as explained here: https://github.com/JuliaParallel/JUHPC?tab=readme-ov-file#2-call-juhpc

Then I must be missing information, as it seems juhpc always needs both directories as arguments to run. And since it checks that they don't exist, how can you reuse an existing JUHPC_SETUP_INSTALLDIR? What would the set of juhpc calls be to create different per-user JULIAUP_INSTALLDIR's, while using the same JUHPC_SETUP_INSTALLDIR?

@omlins omlins reopened this Oct 24, 2024
@omlins
Copy link
Collaborator

omlins commented Oct 24, 2024

@paulmelis I have merged the changes into main. You can now use VERSION=main to try. I will share with you later a concrete use case to hopefully clarify your questions.

@omlins
Copy link
Collaborator

omlins commented Oct 25, 2024

@paulmelis Here is a use case that will hopefully clarify things for you. For a course, we need to use the old supercomputer, because the new one is currently not available for this purpose. The software stack of the old supercomputer is frozen because it is decommissioned very soon. Thus, I have created a Julia HPC setup using JUHPC specifically for this course. I have run juhpc with the following arguments:

JUHPC_SETUP_INSTALLDIR=$SCRATCH/../julia/${HOSTNAME%%[0-9]*}-gpu-nocudaaware/juhpc_setup
JULIAUP_INSTALLDIR="\$SCRATCH/../julia/\$USER/\${HOSTNAME%%[0-9]*}-gpu-nocudaaware/juliaup"

Now all the students can activate the setup by executing (found in this file:

JUHPC_SETUP_INSTALLDIR=$SCRATCH/../julia/${HOSTNAME%%[0-9]*}-gpu-nocudaaware/juhpc_setup

# Load required modules, including correct CPU and GPU target modules (NOTE: the same modules should be loaded when running julia - JUHPC can be used in a module build recipe...)
module load daint-gpu
module switch PrgEnv-cray PrgEnv-gnu
module load cray-hdf5-parallel
module list

# Activate the HPC setup environment variables
. $JUHPC_SETUP_INSTALLDIR/activate

The students never need to run juhpc themselves. When the students call juliaup first time it will install it (and the latest julia version as well). It will automatically install it in the JULIAUP_INSTALLDIR I have chosen above. They can use juliaup and julia just normally. They don't need to know where it was installed, nor what preferences were set.

Does this use case clarify things for you?

@omlins
Copy link
Collaborator

omlins commented Oct 25, 2024

@paulmelis I have released a new version, 0.3.0. Please use this one for your configuration PR!

@omlins
Copy link
Collaborator

omlins commented Oct 31, 2024

@paulmelis Please let me know if the above use case clarified things for you and what do you think could be added to the documentation to make it clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants