Missing parameter value in workflow #358
Comments
|
Thanks for submitting an issue. |
|
Thank you for your interest in Weaver. One thing I notice after deploying the first process is that the output seems to have the incorrect "outputs": {
"staging_output": {
"title": "staging_output",
"formats": [
{
"default": true,
"mediaType": "text/plain"
}
]
}I'm not sure if this can be the cause of the error connecting output of step 1 to input of step 2. I will investigate further in the meantime, but it will take me some time to work around the private docker images. |
|
Follow-up on this issue. I have identified that there is indeed an error introduced by this step: weaver/weaver/processes/wps3_process.py Lines 358 to 364 in 6a7381e A somewhat recent change to the format of results endpoint vs outputs endpoint makes it that the returned content is as follows: {
"staging_output": {
"href": "http://localhost:4002/wpsoutputs/45c45987-8665-4caa-8af8-d90a06608879/gatineau_20170429.tif",
"format": {
"mediaType": "image/tiff"
}
}
}Instead of the previous {
"outputs": {
"staging_output": {
"href": "http://localhost:4002/wpsoutputs/45c45987-8665-4caa-8af8-d90a06608879/gatineau_20170429.tif",
"format": {
"mediaType": "image/tiff"
}
}
}
}The fix will be out soon. |
|
@fmigneault thank you for your quick response! Even having detected a bug, I think you are right in the rest of your comments, so:
|
|
@lhcorralo |
|
Those are great news! Thanks a lot :) |
|
Hi! Just a small question... Is this bug fixed in release 4.3.0? I cannot find it in the changelog, but the dates match, so I am not very sure, sorry... Thanks! |
|
Hi. It is not in |
|
Hi! Bad news, I think the issue is still present :( For easier debugging, I have uploaded https://github.com/lhcorralo/image-utils a simple project, where a docker image may be build. The code is almost trivial :) The workflow is similar to the previous one:
{
"processDescription": {
"process": {
"visibility": "public",
"id": "test_generation",
"title": "Test image generation",
"abstract": "Generates some test GeoTIFF",
"version": "0.0.1",
"outputs": [
{
"id": "output_tifs",
"title": "GeoTIFF Images",
"formats": [
{
"default": true,
"mimeType": "image/tiff"
}
]
}
]
},
"processVersion": "2.0"
},
"immediateDeployment": true,
"executionUnit": [
{
"unit": {
"cwlVersion": "v1.0",
"class": "CommandLineTool",
"baseCommand": [
"generate",
"--output",
"output/"
],
"requirements": {
"DockerRequirement": {
"dockerPull": "image_utils:latest"
}
},
"inputs": {
"base_name": {
"type": "string",
"inputBinding": {
"position": 1,
"prefix": "--base-name"
}
},
"count": {
"type": "int",
"inputBinding": {
"position": 2,
"prefix": "--count"
}
}
},
"outputs": {
"output_tifs": {
"type": {
"type": "array",
"items": "File"
},
"outputBinding": {
"glob": "output/*.tif"
}
}
}
}
}
],
"deploymentProfileName": "http://www.opengis.net/profiles/eoc/dockerizedApplication"
}
{
"processDescription": {
"process": {
"visibility": "public",
"id": "test_blurring",
"title": "Test image blurring",
"abstract": "Test image blurring",
"version": "0.0.1",
"inputs": [
{
"id": "input_files",
"title": "Input images",
"formats": [
{
"mimeType": "image/tiff",
"default": true
}
],
"minOccurs": "1",
"maxOccurs": "unbounded"
}
],
"outputs": [
{
"id": "output_tifs",
"title": "Blurred images",
"formats": [
{
"mimeType": "image/tiff",
"default": true
}
]
}
]
},
"processVersion": "2.0"
},
"immediateDeployment": true,
"executionUnit": [
{
"unit": {
"cwlVersion": "v1.0",
"class": "CommandLineTool",
"baseCommand": [
"-o",
"output/"
],
"requirements": {
"DockerRequirement": {
"dockerPull": "image_utils:latest"
}
},
"inputs": {
"input_files": {
"type": {
"type": "array",
"items": "File"
},
"inputBinding": {
"position": 1,
"prefix": "--image"
}
}
},
"outputs": {
"output_tifs": {
"type": {
"type": "array",
"items": "File"
},
"outputBinding": {
"glob": "output/*.tif"
}
}
}
}
}
],
"deploymentProfileName": "http://www.opengis.net/profiles/eoc/dockerizedApplication"
}
{
"processDescription": {
"process": {
"visibility": "public",
"id": "test_workflow",
"title": "Workflow to test Weaver",
"abstract": "Workflow to test Weaver",
"version": "0.0.1"
}
},
"executionUnit": [
{
"unit": {
"cwlVersion": "v1.0",
"class": "Workflow",
"inputs": {
"base_name": "string",
"count": "int"
},
"outputs": {
"output": {
"type": {
"type": "array",
"items": "File"
},
"outputSource": "blurring/output_tifs"
}
},
"steps": {
"generation": {
"run": "test_generation",
"in": {
"base_name": "base_name",
"count": "count"
},
"out": [
"output_tifs"
]
},
"blurring": {
"run": "test_blurring",
"in": {
"input_files": "generation/output_tifs"
},
"out": [
"output_tifs"
]
}
}
}
}
],
"deploymentProfileName": "http://www.opengis.net/profiles/eoc/workflow"
}
{
"mode": "async",
"response": "document",
"inputs": [
{
"id": "base_name",
"data": "castellon"
},
{
"id": "count",
"data": 4
}
],
"outputs": [
{
"id": "output",
"transmissionMode": "reference"
}
]
}Again, the workflow is failing with the following error: I have attached the weaver.log Thank you! Luis |
Greetings
First of all, congratulations for your great job. I have read the testbeds and used it, and it is an impressive piece of software.
Describe the bug
The execution of a workflow using WPS3 gives the error "ERROR [MainThread][PYWPS] Missing parameter value: input_files", being input_files the input parameter of the second step of the workflow.
How to Reproduce
{ "processDescription": { "process": { "visibility": "public", "id": "Staging_S2L1C", "title": "Sentinel-2 L1C image staging", "abstract": "The test application to deploy docker image and display text", "version": "0.0.1" }, "processVersion": "2.0" }, "immediateDeployment": true, "executionUnit": [ { "unit": { "cwlVersion": "v1.0", "class": "CommandLineTool", "baseCommand": [ "--output", "output/", "--api-endpoint", "http://some-ip:5000/search" ], "requirements": { "DockerRequirement": { "dockerPull": "private.harbor.com/staging_s2l1c:latest" } }, "inputs": { "start_time": { "type": "string", "inputBinding": { "position": 1, "prefix": "--start-time" } }, "end_time": { "type": "string", "inputBinding": { "position": 2, "prefix": "--end-time" } }, "aoi": { "type": "string", "inputBinding": { "position": 3, "prefix": "--aoi" } } }, "outputs": { "staging_output": { "type": { "type": "array", "items": "File" }, "outputBinding": { "glob": "output/*.tif" } } } } } ], "deploymentProfileName": "http://www.opengis.net/profiles/eoc/dockerizedApplication" }{ "processDescription": { "process": { "visibility": "public", "id": "WaterExtent_S2", "title": "RME's Water Extent for Sentinel-2", "abstract": "It gets the water extent given a Sentinel-2 L1C image", "version": "0.0.1", "inputs": [ { "id": "input_files", "title": "Input Sentinel-2 images", "formats": [ { "mimeType": "image/tiff", "default": true } ], "minOccurs": "1" } ], "outputs": [ { "id": "waterextentFiles", "title": "Images with water mask", "formats": [ { "mimeType": "image/tiff", "default": true } ] } ] }, "processVersion": "2.0" }, "immediateDeployment": true, "executionUnit": [ { "unit": { "cwlVersion": "v1.0", "class": "CommandLineTool", "baseCommand": [ "-i", "1", "-o", "output/" ], "requirements": { "DockerRequirement": { "dockerPull": "private.harbor.com/waterextent:0.2.0" } }, "inputs": { "input_files": { "type": { "type": "array", "items": "File" }, "inputBinding": { "position": 1 } } }, "outputs": { "waterextentFiles": { "type": { "type": "array", "items": "File" }, "outputBinding": { "glob": "output/*.tif" } } } } } ], "deploymentProfileName": "http://www.opengis.net/profiles/eoc/dockerizedApplication" }{ "processDescription": { "process": { "visibility": "public", "id": "WorkflowWaterExtent", "title": "Workflow to test Water Extent for Sentinel-2", "abstract": "Workflow to save our beautiful necks", "version": "0.0.1" } }, "executionUnit": [ { "unit": { "cwlVersion": "v1.0", "class": "Workflow", "inputs": { "start_time": "string", "end_time": "string", "aoi": "string" }, "outputs": { "output": { "type": { "type": "array", "items": "File" }, "outputSource": "water_extent/waterextentFiles" } }, "steps": { "staging": { "run": "Staging_S2L1C", "in": { "start_time": "start_time", "end_time": "end_time", "aoi": "aoi" }, "out": [ "staging_output" ] }, "water_extent": { "run": "WaterExtent_S2", "in": { "input_files": "staging/staging_output" }, "out": [ "waterextentFiles" ] } } } } ], "deploymentProfileName": "http://www.opengis.net/profiles/eoc/workflow" }{ "mode": "async", "response": "document", "inputs": [ { "id": "start_time", "data": "2021-07-14" }, { "id": "end_time", "data": "2021-07-20" }, { "id": "aoi", "data": "-0.3 39.8 0.1 40.1" } ], "outputs": [ { "id": "output", "transmissionMode": "reference" } ] }Expected behavior
The workflow should execute properly, returning a mask.
Other comments
Disclaimer
I know, as it is providedd, that the bug cannot be reproduced, as it uses images from private repositories. At the moment I cannot provide it as a easily reproducible example. However, I think the important part of the "bug" is in the process definitions, as the docker images are running properly in local CWL tests.
Exact error
The whole error is
That is, the parametetr failing is the input parameter for the second process of the workflow.
However, the wps:ProcessDescriptions for the process is returning
which shows input_files as input argument.
Step by step executions
The staging process works if not executed as a workflow, and also seems to be working within the workflow (see the log).
Other attempts
I also tried to use
As the whole workflow was working in version 3.5, but the process deployment gives me an error.
Note that I know that the water extent processor returns more than one file, but previous versions of Weaver discards files if more than one file is returned.
Context
Running as hybrid, with manager and worker both running version 4.2.1 from a docker-compose.yaml
The log is attached
log.txt
The text was updated successfully, but these errors were encountered: