Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

beeflow:useContainer support for SquashFS #990

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

beeflow:useContainer support for SquashFS #990

wants to merge 2 commits into from
+91 −11

Conversation

arhall0
Copy link
Collaborator

@arhall0 arhall0 commented Jan 17, 2025

Closes #967

Enables the use of SquashFS format for the DockerRequirement beeflow:useContainer.

I tested this by modifying the clamr example's clamr_wf.cwl file and changed the two DockerRequirements to be:

        DockerRequirement:
            beeflow:useContainer: "$WORKDIR_PATH/clamr-ffmpeg.sqfs"

Where clamr-ffmpeg.sqfs was made using the Dockerfile included with the example.

@arhall0 arhall0 added enhancement New feature or request WIP Work in progress labels Jan 17, 2025
@arhall0 arhall0 self-assigned this Jan 17, 2025
@arhall0 arhall0 linked an issue Jan 17, 2025 that may be closed by this pull request
Add capability to run applications that are Charliecloud sqfs format (DockerRequirement) for tasks #967
Open
@arhall0 arhall0 force-pushed the 967-add-capability-to-run-applications-that-are-charliecloud-sqfs-format-dockerrequirement-for-tasks branch from 401c2c8 to 68c2a51 Compare January 17, 2025 22:10
@arhall0
Copy link
Collaborator Author

arhall0 commented Jan 17, 2025

I'm currently not checking if SquashFS is installed. So an error would be thrown from the job by Charliecloud and end up in the archive. I don't have a good way to check this situation.

We could throw an error earlier with something like if shutil.which('fusermount3') is None then error based on https://hpc.github.io/charliecloud/ch-run.html#image-format

@arhall0 arhall0 requested review from rstyd and pagrubel January 17, 2025 22:25
@arhall0 arhall0 removed the WIP Work in progress label Jan 17, 2025
pagrubel
pagrubel reviewed Jan 28, 2025
View reviewed changes
beeflow/common/crt/charliecloud_driver.py
@@ -101,9 +101,14 @@ def run_text(self, task): # noqa
container_path = '/'.join([self.container_archive, task_container_name]) + '.tar.gz'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
container_path = '/'.join([self.container_archive, task_container_name]) + '.tar.gz'
container_path = '/'.join([self.container_archive, task_container_name]) + '.tar.gz'

Is this line needed anymore?

@pagrubel
Copy link
Collaborator

I'm currently not checking if SquashFS is installed. So an error would be thrown from the job by Charliecloud and end up in the archive. I don't have a good way to check this situation.

We could throw an error earlier with something like if shutil.which('fusermount3') is None then error based on https://hpc.github.io/charliecloud/ch-run.html#image-format

Could this be tested in our github ci, I don't think we have Squash fuse installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pagrubel pagrubel pagrubel left review comments

@rstyd rstyd Awaiting requested review from rstyd

At least 1 approving review is required to merge this pull request.

Assignees

@arhall0 arhall0

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add capability to run applications that are Charliecloud sqfs format (DockerRequirement) for tasks
2 participants
@arhall0 @pagrubel