Criteria for the evaluation of formats

[halfflat]

How can we judge what will be a good format?

Just to start the discussion, I've some criteria that I think are worth considering, broadly divided into two classes.

1. Metadata requirements:

   • How can we identify names or descriptions of measurements across simulator outputs and experimental observations?
     Examples: What does 'cell 7' mean? Where could I find 'segment 18' on this neuronal morphology?

   • How will the format support the aims of open and reproducible science?
     Examples: Where did this data come from? How could I reproduce it? How should I cite it?

   • How will a format cope with the open-ended and heterogeneous metadata practices extant in computational neuroscience and neurophysiology?
     Examples: should software preserve annotations on input data as-is? Whose responsibility is it to align such metadata for the purposes of interoperability? How can it support extensibility for new types of data or metadata?

2. Performance and representation requirements:

   • Is the format suitable for high performance and parallel I/O?

   • Is the format binary portable? (e.g. is it endian agnostic; are floating point values preserved exactly; are structured records represented unambiguously).

   • Is the format self-describing? How much, if any, of the context required to interpret the data is external to the representation?

[fbonnier]

One criterion can be the "popularity" of a file format: is the format already used ? How many tools already use it ? Is there any visualizer or data analyzer that read this format ? Do converter exist to and from this file format ?
This is not a "scientific" criterion, but it can give an idea about the amount of work to spread a new one within the neuron scientists community.

[brenthuisman]

@fbonnier The problem with that way of reasoning is that you'll always end up with what's already popular, and I'm not sure a workgroup discussing potential file standards for Ebrains is required to establish some common formats. We should definitely consider tooling, but not at the cost of what the goal of the working group is: settle on something to improve interoperability and reproducibility. It's a big problem in computational neuroscience, and a platform meant for sharing data (Ebrains) should definitely take a stab at a solution.

[fbonnier]

I am not saying that the most popular one should be the chosen one. I am saying that if a file format is used, it is not because of its name or the number of followers on FB. In general, it's because of its features. An easy example is Google search. It is used because of the amount of features it provides regardless of the attacks on private data.
Perhaps I can ask another question to clarify: what is the file format already used in the majority of the neuroscience community ?
What are its features ?
In the CFD community there is a file format that is open source, and very popular and that provide most of the wanted features: CGNS.
This format is the most used in the entire of CFD community, in private corporations as in public labs. Other formats exist, that have additionnal features, but not as user-friendly and open source than CGNS.
I am wondering if there is such a format in neuroscience.

[brenthuisman]

The neuroscientific formats market a bit more fragmented than that I think. That's why we can make a list of file formats instead of pointing to just the one ;) 'Metaformats' like Sonata, and in a sense also tools like Datalad, I see as an indicator the 'basic file formats' are falling short of something: there apparently is information that is required for reproducing results that cannot be incorporated in the 'basic file format'. Often, if not always, this is a problem of meta-data. When the problem of reproducibility in neuroscience is discussed, this is usually what people refer to: incomplete or un-/ill-structured meta-data that makes it very hard to reproduce a result.

This is separate from the exchange of the data itself, which may very well be a solved problem. It however hinges on your colleague already knowing what you did to create the data.

[laurentperrinet]

Hi!

Thanks for launching this discussion, very timely and needed! We addressed this question recently at our brainhack (with @albertovergani and @JuliaSprenger, see https://github.com/SpikeAI/2020-12_brainhack_Project7) and in particular we tried to make benchmarks to test the performance of two different solutions offered in pyNN: hdf5 and nixio. It appears both have advantages and disadvantages, but that for the moment hdf5 is the most practical in terms of write time and file size.
Importantly, this may evolve in time and since both allow to construct a neo object, we may easily switch at a later time.

In summary, the backend (hdf5 or nix) is one thing but the fact to have the same representation is crucial. And neo seems to be a good candidate. Having tools to retrieve information easily (spike trains of 'cell 7' please?) as you can have in pandas would be a plus.

cheers,
Laurent