Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix remote files handling #648

Merged
merged 12 commits into from
Feb 5, 2025
Merged

Fix remote files handling #648

merged 12 commits into from
Feb 5, 2025

Conversation

LanderOtto
Copy link
Collaborator

@LanderOtto LanderOtto commented Jan 24, 2025

This commit fixes several problems with remote ports.

  • Fixed error introduced in Fixed remote ports #643. When the remote port was a directory, an exception was thrown at translation time
  • CWL SecondaryFile objects were not registered in the DataManager file system
  • Function remap_path did not work correctly when the path was not an URI
  • In CWL, a Directory object has the listing field to list files and directories. These files can exist or not; in the last case, they are created. Before this commit, when a file existed, it was copied twice: once from the copy of the directory and once from the copy of the file itself.
  • Unified the _get_inner_path functions defined in data/manager.py and remotepath.py.
  • Fixed the _get_inner_path method of the RemoteStreamFlowPath class. Now it correctly handles data when they are not mounted from the inner location (e.g., when some input data is already stored inside a target container image).
  • Fixed the registration of files and directories in the DataManager class when created by the relative functions, i.e., write_remote_file and create_remote_directory in the cwl/utils.py module.
  • Improved unit test to check File with secondaryFiles, Directory and literal Directory.

Copy link

codecov bot commented Jan 24, 2025

Codecov Report

Attention: Patch coverage is 77.55102% with 22 lines in your changes missing coverage. Please review.

Project coverage is 71.44%. Comparing base (66080a5) to head (2d5ebdf).
Report is 6 commits behind head on master.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
streamflow/cwl/utils.py 55.10% 15 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #648      +/-   ##
==========================================
+ Coverage   71.37%   71.44%   +0.07%     
==========================================
  Files          88       88              
  Lines       11310    11345      +35     
  Branches     1968     1981      +13     
==========================================
+ Hits         8073     8106      +33     
- Misses       2782     2785       +3     
+ Partials      455      454       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

else:
return path_processor.join(
new_dir,
*os.path.relpath(urllib.parse.unquote(path[7:]), old_dir).split(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the slicing path[7:] in the else branch

@LanderOtto LanderOtto force-pushed the fix/listing branch 2 times, most recently from 2e14fee to 4deafee Compare January 31, 2025 22:08
@LanderOtto LanderOtto marked this pull request as ready for review February 3, 2025 22:44
@GlassOfWhiskey GlassOfWhiskey changed the title Fixed listing Fix remote files handling Feb 5, 2025
@GlassOfWhiskey GlassOfWhiskey merged commit cc58a8b into master Feb 5, 2025
32 checks passed
@GlassOfWhiskey GlassOfWhiskey deleted the fix/listing branch February 5, 2025 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants