[ENH] Proposal for multidimensional array file format

[tyarkoni]

At the BIDS-ComputationalModels meeting, it became pretty clear that a wide range of applications require (or would at benefit considerably from) the ability to read in generic n-dimensional arrays from a binary file. There are at least two major questions that should be discussed here, and then we should move to draft a PR modifying the BIDS-Raw specification:

1. What file format should we use? This should be something generic enough that it can be easily read on all common platforms and languages. The main proposals that came up at the meeting were for numpy (.npy) or HDF5 containers (.h5). While numpy is technically a Python format, it's sufficiently simple and well-supported that there appear to be available libraries for the major languages. Please suggest other potential solutions.

2. How and where should we represent associated metadata? The generic file format (and naming conventions, etc.) will eventually described in the BIDS-Raw spec, alongside all of the other valid formats (.tsv, nifti, etc.). But some applications are likely to require fairly specific interpretations of the data contained in the file. There appears to be some convergence on the notion of representing the relevant metadata in relevant sections of the BIDS-Derivatives spec (or current BEPs)—i.e., every major use case would describe how data in the binary array format should be interpreted when loaded. We could also associate suffixes with use cases, so that a tool like PyBIDS can automatically detect which rules/interpretations to apply at load time. But if there are other proposals (e.g., a single document describing all uses cases), we can discuss that here.

I'm probably forgetting/overlooking other relevant aspects of the discussion; feel free to add to this. Tagging everyone who expressed interest, or who I think might be interested: @JohnGriffiths @maedoc @effigies @yarikoptic @satra.

[effigies]

Note that there are two versions of npy, so compatibility levels of 1 and 2 should be assessed.

My primary concern with npy is that npy is not compressed, npz is just a zip of a directory of npy files, which almost certainly won't handle random read access as well as HDF5.

My primary concern with HDF5 is that it's just a container, and we will find ourselves defining formats. Perhaps just saying it contains only a dataset with name /data or similar will resolve that.

[satra]

in a different world, but probably related to computational models BRAIN has funded development of the NWB standard. to the extent that needs may become similar, it may be worthwhile thinking about supporting NWB in BIDS.

this will make the metadata world both easier (included in the NWB file) and harder (non conformant with BIDS), depending on your point of view. however, the NWB folks are also considering alternatives like exdir, which is like HDF5 but with external metadata and binary blobs as numpy files.

[arokem]

Sorry: could I ask for a bit more context? What kind of data will be stored in these files? If it's large enough to justify parallel processing of its contents, allow me to throw in a plea to consider zarr compatibility. I think that HDF5 could be made to play nice with zarr.

[effigies]

@satra In principle that seems fine, but their HDF5 format looks basically like HDF5 + some mandatory metadata, so if flexibility is a potential downside, it persists.

If it's not a downside, then I have no principled objection.

@arokem The issue driving us here is less the size of the data than the dimensionality. That said, there's no reason that the files couldn't get large enough for random and parallel access to be concerns, which is why I think HDF5 is my inclination (despite my above-noted reservations). The goal is wide interoperability (in particular, C, R, MATLAB and Python) and not reinventing the wheel, so if that format fits, I for one am happy to consider it.

[satra]

@arokem - the NWB folks are also considering zarr compatibility, especially with the N5 API. which would also constrain HDF5, since N5 doesn't support all aspects of it.

[arokem]

Yup. For reference: NeurodataWithoutBorders/pynwb#230

[yarikoptic]

On one hand I am in strong favor of reusing someone else's "schema" and possibly "tooling" on top of HDF5 (container)! NWB might (do not know how well it aligns with the needs of ComputationalModels metadata) be a good choice. Import/export "compatibility" with other computation-oriented formats (like zarr) might be a plus.

BUT thinking in conjunction with 2. -- if we choose a "single file" container format to absorb both (data and metadata), we would step a bit away from "human accessibility" of BIDS. We already have an issue of metadata location duality, e.g. it being present in the data files (nii.gz) headers -- "for machines", and some (often additional but some times just duplicate) in side car files -- "for machines and humans" (related - recent #196). Sure thing bids-validator could assure consistency, but we subconsciously trying to avoid such redundancy, and I wonder if that might still be a way to keep going. May be there is a lightweight format (or some basic "schema" for HDF5) which would not aim to store any possible metadata, but just store minimally sufficient for easy and unambiguous IO of multi-dimensional arrays (if that is the goal here). And then pair it up with the side car .json file convenient access to metadata (defined in BIDS, if there is no existing schema for "ComputationalModels" elsewhere to reuse; not duplicated in the actual data file) for easy human and machines use (without requiring to open the actual data file which would require tooling)? If we end up with a single file format to contain both -- I think we might need to extract/duplicate metadata in a sidecar file anyways for easier human (and at times tools) consumption.

[tyarkoni]

@yarikoptic sorry, I realize on re-read that I wasn't clear, but your proposed approach (putting metadata in the json sidecar and only the raw ndarray in the binary file) is exactly what we seemed to converge on at the end of the BIDS-CM meeting. (I.e., the sidecar would supply the metadata needed to interpret the the common-format array appropriately for the use case specified in the suffix.)

…

On Fri, Apr 5, 2019, 17:29 Yaroslav Halchenko ***@***.***> wrote: On one hand I am in strong favor of reusing someone else's "schema" and possibly "tooling" on top of HDF5 (container)! NWB might (do not know how well it aligns with the needs of ComputationalModels metadata) be a good choice. Import/export "compatibility" with other computation-oriented formats (like zarr) might be a plus. BUT thinking in conjunction with 2. -- if we choose a "single file" container format to absorb both (data and metadata), we would step a bit away from "human accessibility" of BIDS. We already have an issue of metadata location duality, e.g. it being present in the data files (nii.gz) headers -- "for machines", and some (often additional but some times just duplicate) in side car files -- "for machines and humans" (related - recent #196 <#196>). Sure thing bids-validator could assure consistency, but we subconsciously trying to avoid such redundancy, and I wonder if that might still be a way to keep going. May be there is a lightweight format (or some basic "schema" for HDF5) which would not aim to store any possible metadata, but just store minimally sufficient for easy and unambiguous IO of multi-dimensional arrays (if that is the goal here). And then pair it up with the side car .json file convenient access to metadata (defined in BIDS, if there is no existing schema for "ComputationalModels" elsewhere to reuse; not duplicated in the actual data file) for easy human and machines use (without requiring to open the actual data file which would require tooling)? If we end up with a single file format to contain both -- I think we might need to extract/duplicate metadata in a sidecar file anyways for easier human (and at times tools) consumption. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#197 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASjPPjWbCVDMwgkTtTQsKJka3ffwVxeks5vd8AsgaJpZM4cfliR> .

[yarikoptic]

@yarikoptic sorry, I realize on re-read that I wasn't clear, but your proposed approach (putting metadata in the json sidecar and only the raw ndarray in the binary file) is exactly what we seemed to converge on at the end of the BIDS-CM meeting. (I.e., the sidecar would supply the metadata needed to interpret the the common-format array appropriately for the use case specified in the suffix.)

I am delighted to hear that similar minded us independently decided to contribute the XXXX-th model of the wheel to the humanity!

FWIW, I ran into https://news.ycombinator.com/item?id=10858189 on https://cyrille.rossant.net/moving-away-hdf5/ (even @chrisgorgo himself commented on there) -- seems a good number of groups/projects ended up switching from HDF5 to some ad-hoc data blob + metadata files "format". May be it would be worth stating the desired features (I think those weren't mentioned)? e.g. which among the following would be most important?

• portability and library support -- probably a must...
• efficient random access / slicing / ... - desired or not?
  • relates to parallel processing etc. if just a "good to have" then probably not worth jumping to anything fancy
• memory mapping - desired or not?
• compression - desired or not? optional?

or in other words - aiming for processing or archival? if aiming for archival - probably compression is heavily desired... may be could be optional (we already have both .nii and .nii.gz supported IIRC, so could be .blob[.gz])... kinda boils down to .npy - which was also the choice at https://cyrille.rossant.net/moving-away-hdf5/ ;-)

[satra]

@yarikoptic - be careful of that blog post (i think it leads a lot of people astray), and do read all the threads that have emanated from it. for every such use case its easy to point to MATLAB and say that they use it for their base data format. also there are enough posts out there to also say that people who moved away ended up requiring many of the facilities of hdf5 and switching back to it. finally you should take a look at exdir and zarr as well as pointed in earlier threads, and in this followup thread to cyrille's original post and it's comments including the earliest one by konrad hinson (https://cyrille.rossant.net/should-you-use-hdf5/).

at the end of the day it's mostly about blobs and metadata. what one uses to house and link these things is indeed going to keep evolving depending on use cases. so i think the important thing is to think of the use cases, in both short term and to the extent possible longer term.

i like the questions that you have raised, and i think more than the format itself, the thought process should be around those required features, including archiving.

i'm not saying hdf5 is the answer here nor am i saying hdf5 is issue free, but i have also used it through MATLAB and Python over many years, for my use cases, without an issue. i would need to know their specific goals, applications, and use cases to make an informed judgment.

[maedoc]

We've made simple use of HDF5 (often just one or two datasets) for heavy numerical data (well, MB to TBs) in TVB, a computational modeling environment, for the last 7 years without the problems cited in Rossant's blogpost, mainly by keeping usage simple and heavily vetting library usage prior to version changes. I'd expect transparent compression (lz4 has nearly no CPU overhead) and memmapping are particularly useful for BIDS CM.

[effigies]

I've asked the participants in the computational models meeting to contribute their specific use cases, but I'll try to summarize according to my memory.

1. Visual stimuli, which are 2D arrays of luminance/RGB (or similar) values + time. NIfTI has been used to include these, but it's somewhat an abuse of the format.

2. Machine-learning training corpora, which will have an item dimension that will often be shuffled on multiple training runs, and other dimensions that have meaningful structure such as space or time which should be preserved.

3. Simulation state variables. Environment states will look similar to corpora, with some spatial structure, a time dimension, and potentially many runs. Simulated states may or may not be spatially ordered, but still don't fit NIfTI well.

4. Per-ROI covariance matrices. In the general discussion of statistical outputs, per-voxel statistics are easily represented in NIfTI, and even covariance matrices can be packed into dimensions 5 and 6 of NIfTI. For ROI-based outputs, we have the morphometry and timeseries examples to go by for packing single statistics or time series into TSVs, but multiple dimensions per entry would not work easily. We can get around it by having one file per matrix, and that would presumably be an option, but for large numbers of variables or ROIs, a multidimensional array structure would be useful.

I think there were a couple other examples, but as it became clear that some kind of multidimensional array would likely be the result, we did not compile a specific enumeration of all the needed properties, so hopefully we'll get some feedback.

Perhaps @maedoc can clarify the TVB uses that aren't suited to TSV/NIfTI, and what their minimal criteria and additional desiderata are.

[maedoc]

TVB uses that aren't suited to TSV/NIfTI

Surfaces & sparse matrices come to mind; these have straightforward serializations to arrays, so I would specify conventions for the serialization (e.g. faces, triangles, 0-based; sparse CSR, 0-based) instead of worrying about a new format.

[effigies]

Surfaces will be covered in GIFTI. What do you currently use HDF5 for?

[maedoc]

What do you currently use HDF5 for?

We don't use HDF5 for relational metadata, which is stored in an SQL DB and sidecar XML files, but just about everything else.