Migration to Pydra

[omar-rifai]

A rewrite of Nipype is under development under the project name Pydra. Following a discussion with @ghisvail, it could be worthwhile to investigate the advantages/drawbacks of migrating Clinica from Nipype to Pydra. This thread can help take note of the ideas in that direction.

[ghisvail]

This post, and this reply in particular, help clarify the relationship between pydra and the next major version of nipype.

In essence:

• The workflow engine in nipype will be replaced by pydra, which targets more generic data science workflows.
• Integrations with neuroimaging tools will be provided by separate packages unified under the pydra.tasks namespace.
• Containerized distribution of the neuroimaging tools will be provided by neurodocker.
• The umbrella covering these 3 projects above is nipype version 2.

[ghisvail]

Based on this post, plus recent activity from the pydra project on GitHub, it looks like development has converged and stabilized. Besides, the development team is welcoming early feedback and contributions from the neuroimaging community. It's a good opportunity to get some experience first, and later influence the direction of pydra, in order to facilitate its integration with Clinica.

[omar-rifai]

Things to look into (in no particular order):

• What is the status for pydra interfaces for

  • Freesurfer
  • FSL
  • SPM
  • Ants
  • PETPVC
  • MRtrix3
  • Dcm2niix (in PyPI)

• Can a hybrid (Nipype/ Pydra) approach possible

• What about prov in Pydra?

Auditing and provenance tracking: Pydra provides a simple JSON-LD -based message passing mechanism to capture the dataflow execution activties as a provenance graph. These messages track inputs and outputs of each task in a dataflow, and the resources consumed by the task

• Gap analysis
  • How to configure a local workflow to be run in a container
  • How is the slurm integration possible

[ghisvail]

Here is a comprehensive review of our external dependencies and their integration within Clinica, i.e. whether done via a nipype interface, a custom nipype interface or command, or a subprocess call.

Converters

• dcm2niix via subprocess call

T1 linear

• nilearn.image.crop_img via nipype function
• nipype.ants.N4BiasFieldCorrection
• nipype.ants.RegistrationSynQuick

T1 volume

• nipype.spm.NewSegment
• nipype.spm.DARTEL
• nipype.spm.DARTELNorm2MNI
• nipype.spm.Smooth
• ApplySegmentationDeformation via nipype.spm.SPMCommand subclass
• DARTELExistingTemplate via nipype.spm.SPMCommand subclass

Statistics surface

• custom MATLAB script clinicasurfstat.m + embedded copy of the surfstat toolbox + FreeSurfer

  NOTE: Maybe worth investigating replacing it with brainstat which is Python-based.

Statistics volume

• SPM via templated MATLAB scripts + nipype.MATLABCommand

Statistics volume correction

None

PET surface

• nipype.freesurfer.

  • ApplyVolTransform
  • MRIConvert
  • MRISPreproc
  • Tkregister2

• nipype.fsl.Merge

• nipype.petpvc.PETPVC

• nipype.spm.

  • Coregister
  • Normalize12

• freesurfer via subprocess

  • mri_surf2surf
  • mri_vol2surf
  • mris_expand

• freesurfer via nipype.CommandLine

  • gtmseg

• SPM via subprocess

• SPM via custom MATLAB script ApplyInverseDeformationField.m and nipype.MATLABCommand

PET volume

• nipype.spm.
  • Coregister
  • DARTELNorm2MNI
  • Reslice
  • Smooth

Machine-learning

• sklearn
• xgboost

DWI preproc fmap

• niflow.nipype1.workflows.

  • cleanup_edge_pipeline
  • demean_image
  • siemens2rad

• nipype.ants.N4BiasFieldCorrection

• nipype.fsl.

  • ApplyXFM
  • BET
  • Eddy with Cuda
  • FLIRT
  • FUGUE
  • IsotropicSmooth
  • PRELUDE

• nipype.mrtrix3.

  • BrainMask
  • DWIBiasCorrect

DWI preproc T1

• nipype.fsl.

  • BET
  • Eddy with CUDA

• nipype.mrtrix3.DWIBiasCorrect

• ANTs via subprocess

  • antsApplyTransforms
  • CreateJacobianDeterminantImage

DWI connectome

• nipype.freesurfer.MRIConvert

• nipype.fsl.

  • ApplyMask
  • ApplyXFM
  • FLIRT

• nipype.mrtrix.MRTransform

• nipype.mrtrix3.

  • BuildConnectome
  • LabelConvert
  • ResponseSD
  • Tractography
  • EstimateFOD

DWI DTI

• nipype.ants.

  • ApplyTransforms
  • RegistrationSynQuick

• nipype.mrtrix.

  • DWI2Tensor
  • FSL2MRTrix

• nipype.mrtrix3.TensorMetrics

• nipype.fsl.Threshold

EDIT (21/04/2022): update EstimateFOD

[ghisvail]

Updated EstimateFOD in DWI connectome pipeline now using nipype's interface instead of vendoring our own, see #642.

[omar-rifai]

A document to keep track of our progress on these interfaces: https://docs.google.com/spreadsheets/d/1JqOrza0n1mBwS4MpBuGAwZJ_wrpVtHhqU1g9rgh-GuQ/edit?usp=sharing

[djarecka]

Thank you for your work and creating the list. Looks like you are using many various interfaces :) For now we only have some of the FSL, but we can start working on converting more!

Do you have preferences on where to start? Or should we choose something that has small number of interfaces?

[ghisvail]

Do you have preferences on where to start?

I believe it would be great to scan though this list together and make tentative projections of the amount of time and effort required to implement each of those. This could help us sort which pipeline transition could be completed first.

Last time we spoke you mentioned you were doing an open hour with the community, perhaps we could use it for this?

[ghisvail]

We have started our investigation by wrapping what we can with Nipype1Task or using pydra converters where available. We'll see how far we go.