Skip to content

Commit

Permalink
add back pointers to the datasets, other cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
@fedorov
fedorov committed Nov 10, 2018
1 parent f8f2b07 commit 702666b
Show file tree
Hide file tree
Showing 8 changed files with 141 additions and 189 deletions.
14 changes: 12 additions & 2 deletions docs/instructions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,22 @@ All of the test data is also available as a shared folder on Dropbox [here](http

Each participant is asked to complete the following tasks:

1. **Relevant features of the platform**: short narrative of how the platform implements support for the specific type of object, what features are implemented, what features are missing.
1. **Describe the relevant features of the platform**: short narrative of how the platform implements support for the specific type of object, what features are implemented, what features are missing.
2. **Read task**: Demonstrate how the provided test dataset is consumed and presented to the platform user. In this task we aim to evaluate both the accuracy of interpreting the specific test dataset, and the end-user usability.
3. **Write task**: Generate a dataset of the type in question. The dataset will be used to test interoperability of your platform with the other platforms participating in the connectathon.

Your submission **must** include the details about the platform you used to generate the results \(name of the product/platform, version\). If your platform is available publicly, please include access instructions.

**To participate and submit new results**, please use this DICOM4QI Submission Google Form: [http://bit.ly/dicom4qi-submit](http://bit.ly/dicom4qi-submit).
The datasets specific to each of the types of DICOM objects evaluated are described in the following sections.

* [Segmentations](instructions/seg.md)
* [Parametric maps](instructions/pm.md)
* [Image-based measurements](instructions/sr-tid1500.md)
* [Tractography](instructions/tr.md)

**To participate and submit new results**, please use the [DICOM4QI Submission Google Form](http://bit.ly/dicom4qi-submit).

!!! warning
The screenshots and the DICOM objects you submit will be distributed publicly and included in this document in the Results section.

**If you need to update existing content**, please use "edit" icon in the upper right corner of the page, or submit a pull request with changes to this repository on GitHub: [https://github.com/QIICR/DICOM4QI](https://github.com/QIICR/DICOM4QI). Once your PR is merged, the content will be updated automatically.
26 changes: 0 additions & 26 deletions docs/instructions/instructions.md
View file

This file was deleted.

40 changes: 20 additions & 20 deletions docs/instructions/pm.md
View file
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
In this task the participants are expected to demonstrate the capability of the tool to handle loading of the DICOM Parametric Map \(DICOM PM\) object.
!!! info
When ready to submit, use the [DICOM4QI Submission Google Form](http://bit.ly/dicom4qi-submit)

## Tasks for participants
In this task the participants are expected to demonstrate the capability of the tool to handle loading of the [DICOM Parametric Map \(DICOM PM\)](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_A.75.html) object.

1. **Description of the platform/product**:
* **name and version of the software** used for testing
* **free?** if yes - include the download link
* **commercial?** if yes - include the home page for the product
* **open source?** if yes - provide a link to source code
* **what DICOM library do you use?** - if you use certain DICOM toolkit to support this functionality, please list it, if possible
* **Description of the relevant features of the platform**:
* please provide the screenshot of the user interface for the functionality specific to creating/displaying parametric maps
* how do you communicate parametric map semantics to the user \(quantity encoded, units\)?
2. **Read task** \(for each dataset!\)
* load each of the DICOM Parametric map datasets into your platform
* submit a screenshot demonstrating the presentation of the loaded parametric maps to the user by email to Andrey Fedorov
For each of the datasets listed and linked below, please complete the Read task.

Note: the screenshots and the DICOM objects you submit will be distributed publicly and included in this document in the Results section.
!!! info
* all datasets are organized in [this Dropbox folder](https://www.dropbox.com/sh/pnukhsrqtgdgp1n/AAB1vswS7A1c4nIu8wTFnGz_a?dl=0)
* submit the resulting screenshots and datasets by uploading the zip file with the screenshots and resulting objects to this location: https://www.dropbox.com/request/oJFdeFCrRV6nUepxpH1G. Make sure that the file is named to include the name of your platform!

### Test dataset #1
At this time, the only samples of DICOM Parametric map we have were created using [dcmqi](https://github.com/qiicr/dcmqi).

This is a dataset encoding the Apparent Diffusion Coefficient \(ADC\) map produced by a GE scanner as a DICOM Parametric map object that [can be downloaded here](http://slicer.kitware.com/midas3/download/item/257241/paramap.dcm.zip). The original ADC map [available here](http://slicer.kitware.com/midas3/download/item/126196/701-ADCb500.zip) was saved as an object of MR modality by the scanner software.
## Read task

This dataset encodes integer-valued pixels, and the ADC units are micrometers per squared second \(as noted in the object\).
Load each of the DICOM Parametric map datasets into your platform. Submit a screenshot demonstrating the presentation of the loaded parametric maps.

### Test dataset #2
## Write task

This dataset that [can be downloaded here](http://slicer.kitware.com/midas3/download/item/257243/paramap-float.dcm.zip) encodes [the same ADC map as the first dataset](http://slicer.kitware.com/midas3/download/item/126196/701-ADCb500.zip), but in meters per squared second units. The result is an object where each pixel value is less than one. The goal of this object is to test rendering of the true floating point pixels.
Create a DICOM Parametric map object using your platform. Submit as part of your submission package.

## Dataset 1

This is a dataset encoding the Apparent Diffusion Coefficient (ADC) map produced by a GE scanner as a DICOM Parametric map object. The original ADC map available here was saved as an object of MR modality by the scanner software. This dataset encodes integer-valued pixels, and the ADC units are micrometers per squared second (as noted in the object).

## Dataset 2

This dataset encodes the same ADC map as the first dataset, but in meters per squared second units. The result is an object where each pixel value is less than one. The goal of this object is to test rendering of the true floating point pixel values.
90 changes: 22 additions & 68 deletions docs/instructions/seg.md
View file
Original file line number Diff line number Diff line change
@@ -1,88 +1,42 @@
The purpose of this task is to demonstrate support of the DICOM Segmentation Image (DICOM SEG) object.
!!! info
When ready to submit, use the [DICOM4QI Submission Google Form](http://bit.ly/dicom4qi-submit)

The purpose of this task is to demonstrate support of the [DICOM Segmentation Image (DICOM SEG)](http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_A.51.html) object.

The basic read task involves loading the existing DICOM SEG object, and demonstrating segmentation overlay on the image being annotated.

Write task involves volumetric segmentation of a finding (evaluation of the precision/accuracy of the segmentation is out of the scope of this demonstration) and storing the result as a DICOM SEG object.

## Tasks for participants

### Description of the platform/product

* **name and version of the software** used for testing
* **free?** if yes - include the download link
* **commercial?** if yes - include the home page for the product
* **open source?** if yes - provide a link to source code
* **what DICOM library do you use?** - if you use certain DICOM toolkit to support this functionality, please list it, if possible

### Description of the relevant features of the platform

* are both single and multiple segments supported? how are the overlapping segments handled?
* do you support both BINARY and FRACTIONAL segmentation types?
* do you support compressed objects? if yes - for reading, writing, or for both?
* do you render the segment using the color specified in the DICOM object?
* how do you communicate segment semantics to the user?
* how do you support the user in defining the semantics of the object at the time segmentation is created?

### Read task \(for each dataset!\)

* load each of the DICOM SEG datasets that accompany the imaging series into your platform
* submit a screenshot demonstrating the overlay of the segmentation on the CT series, and any other components of the user interface \(e.g., presentation of the ROI semantics to the user, communication of the algorithm metadata\) by email to Andrey Fedorov

### Write tasks

**Single segment**: segment any area \(ideally, the lung lesion in the Test dataset \#1\) using any method available in your platform

**Multiple segments**: segment any two areas in any of the datasets using any method available in your platform \(ideally, such that ther is a single slice where both segments are visible\). Make sure to create separate segment for each of the segmented areas!

* save the result as DICOM SEG; if possible, please include in the series description the name of your tool to simplify comparison tasks!
* run [dciodvfy DICOM validator](http://www.dclunie.com/dicom3tools/dciodvfy.html); iterate on resolving the identified issues as necessary
* as part of quick checks, confirm that the resulting SEG object has the same FrameOfReferenceUID as the source image
* send the resulting objects and the result of **dciodvfy**, explaining any discrepancies found, to Andrey Fedorov by email

Note:

* we are not assessing the accuracy of lesion segmentation, any method is good
* the screenshots and the DICOM SEG objects you submit will be distributed publicly and included in this document in the Results section.

### Datasets

**Test dataset #1**

The imaging dataset is a chest CT with a single lung lesion located in the right lung lobe. This dataset is subject LIDC-IDRI-0314 from The Cancer Imaging Archive \([TCIA](http://www.cancerimagingarchive.net/)\) [LIDC-IDRI](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) collection.

**Image**: Download the zip archive of the CT series [here](http://slicer.kitware.com/midas3/download/item/245513/LIDC-IDRI-0314-CT.zip). The location of the lesion is highlighted below \(visualized in 3D Slicer software\).

<img src="../../images/LIDC-IDRI-0314_screenshot.png">

**Segmentation**: Download the DICOM SEG datasets produced by the platforms that already submitted results [here](http://slicer.kitware.com/midas3/folder/3774) \(data is organized in subfolders corresponding to the individual platforms\).
For each of the datasets listed and linked below, please complete the Read task and the Write task.

**Test dataset #2**
!!! info
* all datasets are organized in [this Dropbox folder](https://www.dropbox.com/sh/6axblmmgrs29oef/AABRvRLu74h9W82QeJa4afmoa?dl=0)

The imaging dataset consists of a PET and CT series for subject QIN-HEADNECK-01-0024 from the TCIA [QIN-HEADNECK](https://wiki.cancerimagingarchive.net/display/Public/QIN-HEADNECK) collection. This data set contains two lesions. This allows to test that the platform can handle more than one segment.
* submit the resulting screenshots and datasets by uploading the zip file with the screenshots and resulting objects to this [Dropbox FileRequests location](https://www.dropbox.com/l/AABQ7CW0pXGQ9VEPbFtCZhCYF1tzcWuKfak). Make sure that the file is named to include the name of your platform!

**Image**: Download the zip archive of the CT series [here](http://slicer.kitware.com/midas3/download/item/245508/QIN-HEADNECK-01-0024-CT.zip), and PET series [here](http://slicer.kitware.com/midas3/download/item/245509/QIN-HEADNECK-01-0024-PET.zip). Lesions are more prominent on the PET series, as shown in the screenshot below \(visualized in 3D Slicer software\).
## Read task

<img src="../../images/QIN-HEADNECK-01-0024_screenshot.png">
Load each of the DICOM SEG datasets that accompany the imaging series into your platform, make a screenshot demonstrating the overlay of the segmentation over the image, and any other components of the user interface (e.g., presentation of the ROI semantics to the user, communication of the algorithm metadata)

**Segmentation**: Download the DICOM SEG datasets produced by the platforms that already submitted results [here](http://slicer.kitware.com/midas3/folder/3786) \(data is organized in subfolders corresponding to the individual platforms\).
## Write task

**Test dataset #3**
!!! note
We are not assessing the accuracy of segmentation, any method is good!

The imaging dataset consists of a PET and CT series for subject QIN-HEADNECK-01-0139 from the TCIA [QIN-HEADNECK](https://wiki.cancerimagingarchive.net/display/Public/QIN-HEADNECK) collection. This data set contains 11 lesions. This allows to test that the platform can handle relatively large number of segments.
**Single segment**: segment any area (ideally, the lung lesion in the Test dataset #1) using any method available in your platform.

**Image**: Download the zip archive of the CT series [here](http://slicer.kitware.com/midas3/download/item/257233/QIN-HEADNECK-01-0139-CT.zip), and PET series [here](http://slicer.kitware.com/midas3/download/item/257234/QIN-HEADNECK-01-0139-PET.zip). Lesions are more prominent on the PET series, as shown in the screenshot below \(visualized in 3D Slicer software\).
**Multiple segments**: segment any two areas in any of the datasets using any method available in your platform (ideally, such that there is a single slice where both segments are visible). Make sure to create separate segment for each of the segmented areas!

<img src="../../images/QIN-HEADNECK-01-0139_screenshot1.png">
<img src="../../images/QIN-HEADNECK-01-0139_screenshot2.png">
Save the result as DICOM SEG. If possible, please include in the series description the name of your tool to simplify comparison tasks!

**Segmentation**: Download the DICOM SEG datasets produced by the platforms that already submitted results [here](http://slicer.kitware.com/midas3/folder/3858) (data is organized in sub-folders corresponding to the individual platforms).
run [`dciodvfy`](http://www.dclunie.com/dicom3tools/dciodvfy.html) DICOM validator; iterate on resolving the identified issues as necessary.

**Test dataset #4**
As part of quick checks, confirm that the resulting SEG object has the same `FrameOfReferenceUID` as the source image.

The imaging dataset consists of an MR series for subject QIN-PROSTATE-001 from the TCIA [QIN-PROSTATE](https://wiki.cancerimagingarchive.net/display/Public/QIN+PROSTATE) collection. This data set contains 1 lesion segmentation, and has non-identity orientation.
## Dataset 1

**Image**: Download the zip archive of the MR series [here](http://slicer.kitware.com/midas3/download/item/257242/701-ADCb500.zip). Lesion is shown in the screenshot below \(visualized in 3D Slicer software\).
The imaging dataset is a chest CT with a single lung lesion located in the right lung lobe. This dataset is subject LIDC-IDRI-0314 from [The Cancer Imaging Archive (TCIA)](https://thecancerimagingarchive.net) [LIDC-IDRI collection](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI).

<img src="../../images/QIN-PROSTATE-001_SEG_screenshot.png">
## Dataset 3

**Segmentation**: Download the DICOM SEG datasets produced by the platforms that already submitted results [here](http://slicer.kitware.com/midas3/folder/3888) (data is organized in subfolders corresponding to the individual platforms).
The imaging dataset consists of a PET and CT series for subject QIN-HEADNECK-01-0139 from the TCIA [QIN-HEADNECK collection](https://wiki.cancerimagingarchive.net/display/Public/QIN-HEADNECK). This data set contains 11 lesions. This allows to test that the implementation can handle more than one segment.
Loading

0 comments on commit 702666b

Please sign in to comment.