Automation of ro-crate workflow imports

[hexylena]

xref discussions in various channels with @fbacall @supernord Carole across 4 different platforms and private and public channels :)

Our Problem

We want our workflow RO-Crates ingested regularly, kept up to date from their source repository (https://github.com/galaxyproject/training-material) where we generate the crates on the fly from the .ga files that are in our repo.

I'll put my proposed solutions as threads within this discussion, and anyone can start a thread too of course, or reply to those individual threads of discussion to keep everything organised.

Final Comments

The "push" version in the end would probably take less time to implement, but, we'd rather "pull" because we feel it's a more general slution that would then work for more organisations that could maybe add sitemap.xml URLs to their organisational profile

[hexylena]

Solution: Pull

This is our ideal, as we (the GTN) wouldn't have to write another line of code. Here the WorkflowHub does like TeSS, and scrapes our sitemap.xml, looks for anything that looks like a .zip or .tgz and attempts to import it if it's a valid RO-Crate, and adds them to a specific organisation (or can we write that in the ro-crate in a way that it gets ingested/interpreted/automatically added to the right organisation?)

And then the WorkflowHub also thus handles things like repeatedly importing an unchanging RO-Crate and just not issuing a new version unless it's changed, and maintaining the mapping of (this url to an ro-crate) → this wfhub workflow id

Implementation Suggestions

Perhaps this would be implemented by WFHub allowing you to add a url to a sitemap.xml to your profile, or organisational profile, and then RO-Crates discovered from that URL would be automatically imported and associated with your organisation/profile.

[hexylena]

If y'all went this route, we would happily mark our workflows up with the appropriate bioschemas profile (https://bioschemas.org/profiles/ComputationalWorkflow/1.0-RELEASE), it'd just be up to WFHub to behave exactly like TeSS and scrape + import. I think it'd be a really nice complement: TeSS for learning resources + events, WFHub for Workflows, all from standard bioschemas json-ld

[fbacall]

Apologies for not replying to this sooner.

My concern with this approach would be that its fairly uncommon for workflows to have a webpage where metadata could be embedded (like in GTN), unlike with events and training materials.

[hexylena]

Hey @fbacall thanks for the reply! Yeah, ok, that makes sense, sadly. I appreciate the explanation of rationale!

[hexylena]

Solution: Push

Alternatively we are looking for an answer on whether or not we can repeatedly upload the same RO-Crate (.zip) without issuing new versions of workflows. i.e. running the following code in a loop, will it generate new versions, or will wfhub (hopefully) reject the duplicate upload and not issue a new version

curl -X POST -H "Authorization: Token YOUR_TOKEN_HERE" \
     -F workflow[project_ids][]=1234 \
     -F ro_crate=@my_ro_crate.crate.zip https://workflowhub.eu/workflows

Then we will just write a loop that attempts uploading repeatedly and we can set that to automate forever and forget about it :)

[supernord]

Thanks @hexylena 😄
Work to date on Export ro-crate-metadata.json for building RO Crates of the Workflows can be found here: galaxyproject/training-material#4271

[supernord]

@uwwint

[supernord]

galaxyproject/training-material#4271 now merged

Some further notes:

• need to extract version from *.ga file and embed in RO-crate metadata
• push model would be ideal (as in comments above)
• GTN to decide if, and when, a new submission to WorkflowHub happens (based on changing versions)
• WorkflowHub to take version number and assign this to the workflow entry
• @supernord to map WorkflowHub fields to available GTN metadata

@fbacall / @uwwint

[hexylena]

So we already have the version number in the ro-crate-metadata, do you need it there differently?

  "@id": "main_workflow.ga",                                      
  "@type": [                                                      
    "File",                                                       
    "SoftwareSourceCode",                                         
    "ComputationalWorkflow"                                       
  ],                                                              
  "author": [                                                     
      # TODO, will fix this.
  ],                                                              
  "license": {                                                    
    "@id": "https://spdx.org/licenses/CC-BY-4.0"                  
  },                                                              
  "name": "Genome Assembly of MRSA using Illumina MiSeq Data",    
  "version": 3,                                                   
  "programmingLanguage": {                                        
    "@id": "https://w3id.org/workflowhub/workflow-ro-crate#galaxy"
  }                                                               
},

The version in the .ga file is maybe not representative of all true internal versions which is why we calculate the version correctly in the .zip file we create and attach it to the ro-crate-metadata.json

push model would be ideal (as in comments above)

pull is still my ideal! but we'll find a way to make it work

GTN to decide if, and when, a new submission to WorkflowHub happens (based on changing versions)

oh, this is really too bad then. I guess the API endpoint isn't idempotent? we'll find a way but it's some additional complexity.
Is there an API endpoint I can use to get "what is the latest version that WFHub knows about?

[hexylena]

here is an example of the current output, the missing author is a known issue, will resolve as soon as we can. https://training.galaxyproject.org/training-material/api/workflows/assembly-mrsa-illumina/main_workflow/rocrate.zip