From eeff2e056936e1e6fcc7ff52e310949439b52688 Mon Sep 17 00:00:00 2001 From: sinolonghai Date: Fri, 27 Sep 2024 18:46:09 -0400 Subject: [PATCH 1/6] Update qchem.md Update slurm script and add BrianQC contents. --- docs/Documentation/Applications/qchem.md | 85 +++++++++++++++--------- 1 file changed, 55 insertions(+), 30 deletions(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index 792a956fe..99e4aacd5 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -4,61 +4,86 @@ ## Running Q-Chem -The `q-chem` module should be loaded to set up the necessary environment. The `module help` output can provide more detail. In particular, the modulefile does not set the needed environment variable `QCSCRATCH`, as this is likely unique for each run. `QCLOCALSCR` is set by default to `/tmp/scratch`, but one may wish to point to a more persistent location if files written to local scratch need to be accessed after the job completes. Users can easily do this in their Slurm scripts or at the command line via `export` (Bash) or `setenv` (csh). +The `q-chem` module should be loaded to set up the necessary environment. The `module help` output can provide more detail. In particular, the modulefile does not set the needed environment variable `QCSCRATCH`, as this is likely unique for each run. Users should do this in their Slurm scripts or at the command line via `export` (bash) or `setenv` (csh). The simplest means of starting a Q-Chem job is via the supplied `qchem` wrapper. The general syntax is: `qchem -slurm <-nt number_of_OpenMP_threads> ` -For example, to run a job with 36 threads: +For example, to run a job with 104 threads: -`qchem -slurm -nt 36 example.in` +`qchem -slurm -nt 104 example.in` !!! tip "Note" The Q-Chem input file must be in the same directory in which you issue the qchem command. In other words, `qchem ... SOMEPATH/` won't work. -For a full list of which types of calculation are parallelized and the types of parallelism, see the [Q-Chem User's Manual](https://manual.q-chem.com/5.3/). +For a full list of which types of calculation are parallelized and the types of parallelism, see the [Q-Chem User's Manual](https://manual.q-chem.com/6.2/). To save certain intermediate files for, *e.g.*, restart, a directory name needs to be provided. If not provided, all scratch files will be automatically deleted at job's end by default. If provided, a directory `$QCSCRATCH/savename` will be created and will hold saved files. In order to save all intermediate files, you can add the `-save` option. -A template Slurm script to run Q-Chem with 36 threads is: +A template Slurm script to run Q-Chem with 104 threads is: -??? example "Sample Submission Script" +### Sample Submission Script for Kestrel - ```bash +``` + #!/bin/bash #SBATCH --job-name=my_qchem_job #SBATCH --account=my_allocation_ID - #SBATCH --ntasks=36 - #SBATCH --time=01:00:00 + #SBATCH --nodes=1 + #SBATCH --tasks-per-node=104 + #SBATCH --time=01:00:00 + #SBATCH --exclusive #SBATCH --mail-type=BEGIN,END,FAIL #SBATCH --mail-user=your_email@domain.name #SBATCH --output=std-%j.out #SBATCH --error=std-%j.err # Load the Q-Chem environment - module load q-chem - - # Go to the location of job files, presumably from where this file was submitted - cd $SLURM_SUBMIT_DIR - - # Set up scratch space - SCRATCHY=/scratch/$USER/${SLURM_JOB_NAME:?} - if [ -d $SCRATCHY ] - then - rm -r $SCRATCHY + module load q-chem/6.2 + + if [ -e /dev/nvme0n1 ]; then + SCRATCH=$TMPDIR + echo "This node has a local storage and will use $SCRATCH as the scratch path" + else + SCRATCH=/scratch/$USER/$SLURM_JOB_ID + echo "This node does not have a local storage drive and will use $SCRATCH as the scratch path" fi - mkdir -p $SCRATCHY - export QCSCRATCH=$SCRATCHY - - # Move files over - cp * $SCRATCHY/. - cd $SCRATCHY - - # Start run. Keep restart files without intermediate temp files in directory called "my_save" - qchem -nt 36 job.in job.out my_save - ``` -To run this script on Swift, the number of threads can be changed to 64. + mkdir -p $SCRATCH + + export QCSCRATCH=$SCRATCH + export QCLOCALSCR=$SCRATCH + + jobnm=qchem_test + + if [ $SLURM_JOB_NUM_NODES -gt 1 ]; then + QCHEMOPT="-mpi -np $SLURM_NTASKS" + else + QCHEMOPT="-nt $SLURM_NTASKS" + fi + + echo Running Q-Chem with this command: qchem $QCHEMOPT $jobnm.com $jobnm.out + qchem $QCHEMOPT $jobnm.com $jobnm.out + + rm $SCRATCH/* + rmdir $SCRATCH +``` + +To run this script on HPCs other than Kestrel, the number of threads should be changed accordingly. A large number of example Q-Chem input examples are available in `/nopt/nrel/apps/q-chem//samples`. +## Running BrianQC +BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs, which is significantly faster for some larger ab initio jobs. BrianQC uses the same input file as Q-Chem. To run BrianQC, please make the following changes to the sample Slurm scipt above: +1. Add this line in the header section: "#SBATCH --gres=gpu:1" +2. Load the BrianQC module instead of Q-Chem: "module load brianqc" +3. Add "-gpu" in $QCHEMOPT like: + ``` + if [ $SLURM_JOB_NUM_NODES -gt 1 ]; then + QCHEMOPT="-gpu -mpi -np $SLURM_NTASKS" + else + QCHEMOPT="-gpu -nt $SLURM_NTASKS" + fi + ``` +4. Submit jobs through the GPU login nodes on Kestrel or add "#SBATCH -p gpu" to the header of slurm file if running on Swift. + From 113482b7fb921802827bf7690fae53e514aaef4b Mon Sep 17 00:00:00 2001 From: Haley Yandt <46908710+yandthj@users.noreply.github.com> Date: Tue, 1 Oct 2024 08:44:55 -0600 Subject: [PATCH 2/6] Update docs/Documentation/Applications/qchem.md --- docs/Documentation/Applications/qchem.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index 99e4aacd5..df6bd146d 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -74,7 +74,7 @@ To run this script on HPCs other than Kestrel, the number of threads should be c A large number of example Q-Chem input examples are available in `/nopt/nrel/apps/q-chem//samples`. ## Running BrianQC -BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs, which is significantly faster for some larger ab initio jobs. BrianQC uses the same input file as Q-Chem. To run BrianQC, please make the following changes to the sample Slurm scipt above: +BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs, which is significantly faster for some larger ab initio jobs. BrianQC uses the same input file as Q-Chem. To run BrianQC, please make the following changes to the sample Slurm script above: 1. Add this line in the header section: "#SBATCH --gres=gpu:1" 2. Load the BrianQC module instead of Q-Chem: "module load brianqc" 3. Add "-gpu" in $QCHEMOPT like: From d34161b41e132c2c36a30d87a8a5f56ef787fa2e Mon Sep 17 00:00:00 2001 From: Haley Yandt <46908710+yandthj@users.noreply.github.com> Date: Tue, 1 Oct 2024 08:45:04 -0600 Subject: [PATCH 3/6] Update docs/Documentation/Applications/qchem.md --- docs/Documentation/Applications/qchem.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index df6bd146d..9f4ba5d48 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -85,5 +85,5 @@ BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs QCHEMOPT="-gpu -nt $SLURM_NTASKS" fi ``` -4. Submit jobs through the GPU login nodes on Kestrel or add "#SBATCH -p gpu" to the header of slurm file if running on Swift. +4. Submit jobs through the GPU login nodes on Kestrel or add "#SBATCH -p gpu" to the header of the batch script if running on Swift. From 2de2d72743b4d60432294459d187b7910aab12c3 Mon Sep 17 00:00:00 2001 From: Haley Yandt <46908710+yandthj@users.noreply.github.com> Date: Tue, 1 Oct 2024 08:45:14 -0600 Subject: [PATCH 4/6] Update docs/Documentation/Applications/qchem.md --- docs/Documentation/Applications/qchem.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index 9f4ba5d48..aeee18c05 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -69,7 +69,7 @@ A template Slurm script to run Q-Chem with 104 threads is: rmdir $SCRATCH ``` -To run this script on HPCs other than Kestrel, the number of threads should be changed accordingly. +To run this script on systems other than Kestrel, the number of threads should be changed accordingly. A large number of example Q-Chem input examples are available in `/nopt/nrel/apps/q-chem//samples`. From 253b66ba9e803aa6636ef593e703a0c768cf5362 Mon Sep 17 00:00:00 2001 From: sinolonghai Date: Fri, 8 Nov 2024 13:36:25 -0500 Subject: [PATCH 5/6] Update qchem.md --- docs/Documentation/Applications/qchem.md | 49 ++++++++++++++++++++---- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index aeee18c05..bed53dbee 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -74,16 +74,51 @@ To run this script on systems other than Kestrel, the number of threads should b A large number of example Q-Chem input examples are available in `/nopt/nrel/apps/q-chem//samples`. ## Running BrianQC -BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs, which is significantly faster for some larger ab initio jobs. BrianQC uses the same input file as Q-Chem. To run BrianQC, please make the following changes to the sample Slurm script above: -1. Add this line in the header section: "#SBATCH --gres=gpu:1" -2. Load the BrianQC module instead of Q-Chem: "module load brianqc" -3. Add "-gpu" in $QCHEMOPT like: - ``` +BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs, which is significantly faster for some larger ab initio jobs. BrianQC uses the same input file as Q-Chem. Below is a sample slurm script for BrianQC, which should be submitted on the GPU login nodes of Kestrel. If running on Swift, please also add "#SBATCH -p gpu" to the header of this script. +``` + #!/bin/bash + #SBATCH --job-name=my_qchem_job + #SBATCH --account=my_allocation_ID + #SBATCH --nodes=1 + #SBATCH --tasks-per-node=104 + #SBATCH --time=01:00:00 + #SBATCH --gres=gpu: + #SBATCH --mem= + #SBATCH --mail-type=BEGIN,END,FAIL + #SBATCH --mail-user=your_email@domain.name + #SBATCH --output=std-%j.out + #SBATCH --error=std-%j.err + + # Load the Q-Chem environment + module load brianqc + + if [ -e /dev/nvme0n1 ]; then + SCRATCH=$TMPDIR + echo "This node has a local storage and will use $SCRATCH as the scratch path" + else + SCRATCH=/scratch/$USER/$SLURM_JOB_ID + echo "This node does not have a local storage drive and will use $SCRATCH as the scratch path" + fi + + mkdir -p $SCRATCH + + export QCSCRATCH=$SCRATCH + export QCLOCALSCR=$SCRATCH + + jobnm=qchem_test + if [ $SLURM_JOB_NUM_NODES -gt 1 ]; then QCHEMOPT="-gpu -mpi -np $SLURM_NTASKS" else QCHEMOPT="-gpu -nt $SLURM_NTASKS" fi - ``` -4. Submit jobs through the GPU login nodes on Kestrel or add "#SBATCH -p gpu" to the header of the batch script if running on Swift. + + echo Running Q-Chem with this command: qchem $QCHEMOPT $jobnm.com $jobnm.out + qchem $QCHEMOPT $jobnm.com $jobnm.out + + rm $SCRATCH/* + rmdir $SCRATCH +``` + + From fd16e22654ee61a137ef449cbcbcf6e3977649f7 Mon Sep 17 00:00:00 2001 From: sinolonghai Date: Mon, 16 Dec 2024 15:10:04 -0500 Subject: [PATCH 6/6] Update qchem.md --- docs/Documentation/Applications/qchem.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Documentation/Applications/qchem.md b/docs/Documentation/Applications/qchem.md index bed53dbee..d20df6603 100644 --- a/docs/Documentation/Applications/qchem.md +++ b/docs/Documentation/Applications/qchem.md @@ -28,7 +28,7 @@ A template Slurm script to run Q-Chem with 104 threads is: ``` #!/bin/bash #SBATCH --job-name=my_qchem_job - #SBATCH --account=my_allocation_ID + #SBATCH --account=[my_allocation_ID] #SBATCH --nodes=1 #SBATCH --tasks-per-node=104 #SBATCH --time=01:00:00 @@ -78,12 +78,12 @@ BrianQC is the GPU version of Q-Chem and can perform Q-Chem calculations on GPUs ``` #!/bin/bash #SBATCH --job-name=my_qchem_job - #SBATCH --account=my_allocation_ID + #SBATCH --account=[my_allocation_ID] #SBATCH --nodes=1 #SBATCH --tasks-per-node=104 #SBATCH --time=01:00:00 - #SBATCH --gres=gpu: - #SBATCH --mem= + #SBATCH --gres=gpu:[number of gpu] + #SBATCH --mem=[requested memory] #SBATCH --mail-type=BEGIN,END,FAIL #SBATCH --mail-user=your_email@domain.name #SBATCH --output=std-%j.out