Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Remove invalid handling of 'data' output as if it was a valid File reference #185

Closed
fmigneault opened this issue Jul 13, 2020 · 1 comment
Assignees
Labels
feature/CWL Issue related to CWL support feature/job Issues related to job execution, reporting and logging. project/CRIM-DEVOPS Project linked to the CRIM project DevOps/Weaver (https://crim-ca.atlassian.net/browse/GD-47). project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) triage/enhancement New feature or request triage/feature New requested feature.

Comments

@fmigneault
Copy link
Collaborator

fmigneault commented Jul 13, 2020

Inside weaver.processes.wps_package.WpsPackage.make_location_outputs, some handling of non-File type resulting from CWL execution is still being done. (backward compatibility)

result_loc = cwl_result[output_id]
result_wps = os.path.join(wps_out_dir, os.path.split(result_loc)[-1])
# FIXME: instead raise error for non-file output, (don't allow hacking output using string type)?
# if any process provides an explicit URL, it should be handled by itself (we don't rewrite)
if os.path.isfile(str(result_loc)):
self.logger.warning("[DEPRECATED] Process output '%s' defined as CWL type other than File will "
"be unsupported in a future release.", output_id)
if os.path.realpath(result_loc) != os.path.realpath(result_wps):
self.logger.info("Moving: [%s] -> [%s]", result_loc, result_wps)
shutil.move(result_loc, result_wps)
self.response.outputs[output_id].data = result_wps
self.logger.info("Resolved WPS output [%s]: [%s]", output_id, result_wps)

This allows output 'string' type to refer to some actual file as literal data, but makes it hard to maintain and predict behavior of the application execution as we still need to adjust internal paths in order to make it available on WPS-outputs URL. It is even more complicated when an actual string should be returned from a given process (which action to take cannot be guaranteed).

To motivate correct writing of CWL application packages, we should enforce using File for such references. A string will simply be returned as is, just like any other literal data output.

Relates to DAC-533

@fmigneault fmigneault added triage/enhancement New feature or request triage/feature New requested feature. feature/CWL Issue related to CWL support feature/job Issues related to job execution, reporting and logging. labels Jul 13, 2020
@fmigneault fmigneault self-assigned this Jul 13, 2020
fmigneault added a commit that referenced this issue Jul 14, 2020
@fmigneault fmigneault added project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) project/CRIM-DEVOPS Project linked to the CRIM project DevOps/Weaver (https://crim-ca.atlassian.net/browse/GD-47). labels May 7, 2024
@fmigneault
Copy link
Collaborator Author

fmigneault commented Sep 12, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/CWL Issue related to CWL support feature/job Issues related to job execution, reporting and logging. project/CRIM-DEVOPS Project linked to the CRIM project DevOps/Weaver (https://crim-ca.atlassian.net/browse/GD-47). project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) triage/enhancement New feature or request triage/feature New requested feature.
Projects
None yet
Development

No branches or pull requests

1 participant