Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Harden cut command in Snakefiles #5288

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Harden cut command in Snakefiles #5288

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions share/picongpu/examples/LaserWakefield/lib/python/snakemake/Snakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,13 @@ rule simulate:
sbatch {input}/tbg/submit.start > simulated/job_id_{params.name}.txt
fi

job_id=$(cut -c 21-27 simulated/job_id_{params.name}.txt)
job_id=$(cut -d' ' -f4 simulated/job_id_{params.name}.txt)
status=$(squeue --jobs $job_id -o "%2t" | tail -n 1 )

# if job isn't running, pending or configuring -> restart job
if ! [[ "$status" == "PD" || "$status" == "R " || "$status" == "CF" ]]; then
sbatch {input}/tbg/submit.start > simulated/job_id_{params.name}.txt
job_id=$(cut -c 21-27 simulated/job_id_{params.name}.txt)
job_id=$(cut -d' ' -f4 simulated/job_id_{params.name}.txt)
status=$(squeue --jobs $job_id -o "%2t" | tail -n 1 )
fi
# wait till job is finished
Expand Down
4 changes: 2 additions & 2 deletions share/picongpu/examples/LaserWakefield/lib/python/snakemake/Snakefile_LSF
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,13 @@ rule simulate:
bsub {input}/tbg/submit.start > simulated/job_id_{params.name}.txt
fi

job_id=$(cut -c 6-12 simulated/job_id_{params.name}.txt)
job_id=$(cut -d'<' -f2 simulated/job_id_{params.name}.txt | cut -d'>' -f1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arthur told me, that the cut -d' ' -f4 command worked on hemera. Why did you choose cut -d'<' -f2 here? What is the difference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -d sets the delimiter (to whitespace resp. </> in this case) and the -f selects the "field", i.e., the colum with respect to this delimiter (indexing starts with 1). So, cut -d' ' -f4 means: Split the string with respect to whitespaces and select the fourth field resulting from that.

The LSF version: My best guess was that the output from bsub is as shown in this stackoverflow answer. The two cuts split the string into

Full: "Job <930> is submitted to default queue <normal>."
        1  ^          2                        ^   3
First cut: "XXX> ..."
           1   ^   2
Second cut: "XXX"

Could have been a regex but this was faster.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This seems very reasonable.

status=$(bjobs -noheader -o stat:4 $job_id)

# if job isn't running or pending -> restart job
if ! [[ "$status" == "PEND" || "$status" == "RUN " ]]; then
bsub {input}/tbg/submit.start > simulated/job_id_{params.name}.txt
job_id=$(cut -c 6-12 simulated/job_id_{params.name}.txt)
job_id=$(cut -d'<' -f2 simulated/job_id_{params.name}.txt | cut -d'>' -f1)
status=$(bjobs -noheader -o stat:4 $job_id)
fi
# wait till job is finished
Expand Down