Reconsidering the BIDS conversion workflow

[ghisvail]

This discussion thread explores alternative converter workflows based on user experience within ARAMIS, evolution of the BIDS specification and feedback from interacting with external source providers such as LONI or XNAT.

[ghisvail]

Imaging Data

Imaging data are downloaded as a bundle of one or mutliple archives containing a flat or nested file hierarchy of DICOM and / or NIfTI files. Paths and filenames will contain some of means of identification, such as a single image data ID for LONI datasets or a compound ID for OASIS-3 or HABS. LONI datasets provide a collection manifest linking the image data ID with imaging session information. OASIS-3 and HABS compound ID encodes session information directly.

Clinica Data

Clinical data are the largest source of variance between converters. They are usually provided as tabular data saved in Excel or CSV format with different encoding (UTF-8, signed UTF-8, ...). Clinical data may be bundled as a single large file (NIFD, UK Biobank) or split between multiple files (ADNI, AIBL, HABS, OASIS-3, ...). Linking of the clinical data with imaging sessions may be explicit (LONI, HABS) or need to be computed as part of a normalization process (OASIS-3).

[omar-rifai]

Hello @ghisvail, thanks for starting this thread. Can you add a couple of points (in bullet form) you think ought to change from the current way things are handled. Let's try to keeps this short and to the point linking to the appropriate pages (e.g BIDS specs) when necessary.

[ghisvail]

I have added a summary of the specs below with appropriate links to the spec. The aim is to list what we currently have (this post), what the current state of the BIDS spec is (the post below) and to brainstorm possible improvements (post to come below).

[ghisvail]

A quick reminder of the fundamentals of the BIDS specification (version 1.6.0 at this time).

Minimal requirements

The bare minimum is to provide the imaging data (rawdata in BIDS terms) as NIfTI following a normalized naming structure. The root of the rawdata must contain a dataset description file. Other requirements are modality specific.

Additional recommendations

Besides the requirements above, the BIDS specification prescribes a number of recommendation which may or may not be followed.

A non-exhaustive list of those recommendations include:

• Providing or referencing the dataset used to generate the rawdata. Those are named sourcedata in the BIDS specification.
• Providing derived data processed from the rawdata. Those are named derivatives in the BIDS specification.
• Organizing a dataset as separate sourcedata, rawdata and derivatives folders, instead of nesting sourcedata and derivatives within rawdata.
• Generating participant-level, session-level and scan-level manifests with query attributes in tabular form.
• Providing phenotyped or survey data under a separate phenotype folder in tabular form.
• Providing modality-specific metadata in a separate sidecar JSON file.
• Providing attribute metadata for tabular data in a separate sidecar JSON file.

And many more.

[ghisvail]

LONI datasets

Regarding the LONI datasets (ADNI, AIBL, NIFD), imaging data collections already provide enough information to achieve at least the minimal requirements. The collection metadata (downloadable in CSV format) contains a manifest of the image data ID linked to the respective participant and session identifiers, plus some basic query metadata (group, sex, age, acquisition date). Those metadata can also be used to generate a minimal participant, session and scan manifests.

Using collections as input for image data conversion would also avoid issues like #419 where the clinical and imaging are out-of-sync, resulting in empty or partial processing.

Clinical data can still be useful to filter imaging data, but processing should be made robust against reliability issues. In retrospect, it makes sense to be careful of coherance issues between the clinical and imaging data since both are collected and maintained separately typically.