Directory structure organization and file naming convention

[SylvainTakerkart]

This issue is where we'll discuss the directory structure that will contain the data and metadata, the organization of the directory and sub-directories, the number of levels in the hierarchy of the directories, the naming of the directories and sub-directories, the naming of the files...

The different elements that might be included in this hierarchy are: the experiment itself, the subject, the recording session etc.

[SylvainTakerkart]

We can of course (and should) get inspiration from how BIDS do things for human neuroimaging. Here is an example from the BIDS main webpage (on the right handside): https://bids.neuroimaging.io/assets/img/dicom-reorganization-transparent-white_1000x477.png . But we obviously might deviate from this!

[chrisvdt]

I would like to suggest the following hierarchy:
Project_name /Dataset_name /subject_name /date_time_session /raw (versus /derived)
within such a folder, all files associated with the same session could have the same ID , which makes it easy to automate retrieval of files.

[jbpoline]

Learning from the BIDS issues to not repeat them would be good as well, there is a project to have a json schema to describe BIDS that I think should be used as a model, to avoid the inconsitencies that we have in BIDS. This PR is pointing to the json schema code in BIDS

[SylvainTakerkart]

I would like to suggest the following hierarchy:
Project_name /Dataset_name /subject_name /date_time_session /raw (versus /derived)
within such a folder, all files associated with the same session could have the same ID , which makes it easy to automate retrieval of files.

thanks for this suggestion!

an interesting (and necessary) topic we'll have to discuss, will be to come to an agreement on what is a "project", what is a "dataset" etc. ; from all our discussions with the experimenters (mostly electrophysiologists working on non-human primate and rodents) within our institute, this was not as simple as one could expect ;)

[chrisvdt]

I would like to suggest the following hierarchy:
Project_name /Dataset_name /subject_name /date_time_session /raw (versus /derived)
within such a folder, all files associated with the same session could have the same ID , which makes it easy to automate retrieval of files.

thanks for this suggestion!

an interesting (and necessary) topic we'll have to discuss, will be to come to an agreement on what is a "project", what is a "dataset" etc. ; from all our discussions with the experimenters (mostly electrophysiologists working on non-human primate and rodents) within our institute, this was not as simple as one could expect ;)

Yes, I can understand this, we've also had our discussions about this. So a bit of explanation. For us Project is like the introduction to an article, it covers the background or your research, affiliations, authors, hypotheses. and related work.
Within a project you might run different experiments, using different methods, with different sets of animals. So here we think you should define different datasets. Datasets prescribe how data should be preprocessed, which is determined by which subjects were included, and what methods were used. This should be consistent within a dataset.
Below this level could be more levels, but at the least I think there should be a Subject level and a Date_time_session level.

I've included a distinction between raw and derived, because when you share data you could decide only to share raw or only derived, and having this separation in the basis would make this easier. (I think our aim should be to make data more accessible and shareable.)

[SylvainTakerkart]

thanks for your explanations!

Project_name /Dataset_name /subject_name /date_time_session /raw (versus /derived)

We're actually very close to what you're suggesting in what we're locally trying to implement... Here is what we're suggesting for now:
exp-NAME/sub-GUID/ses-YYYYMMDD_XXX_BBBB/rawdata
exp-NAME/sub-GUID/ses-YYYYMMDD_XXX_BBBB/derivatives
exp-NAME/sub-GUID/ses-YYYYMMDD_XXX_BBBB/metadata

Interestingly, we did not use the concepts of Project and Dataset, but we used the concept of Experiment, with one less level of hierarchy compared to what you describe... Our specs are here: https://int-nit.github.io/AnDOChecker/ . And we have the equivalent of the BIDS validator running for these specs: https://andocheck.int.univ-amu.fr/ . But we're still considering this is under development (we only have a few beta testers internally for now...) so we're totally up for merging / fusing / adapting / tweeking, which is what we hope can happen with this discussion!!!

Some details:

• we're pushing towards using a GUID as subject name (which is a bit difficult for investigators working with one-three monkeys, but is more easily acceptable for rodent-based research);
• to characterize the session, we also use the date, but XXX is a user defined session number (where you use the Time); and we left BBBB as an formatless string so that users can add what they want...

[SylvainTakerkart]

pinging @satra @yarikoptic @bendichter: you guys made the chose to go flatter (instead of using several levels of hierarchy as in BIDS and as the two aforementioned suggestions) with dandi, right? could you quickly summarize why?

[SylvainTakerkart]

pinging @tgbugs; could you also tell us what you came up with for sparc?

[SylvainTakerkart]

pinging @lepmik; also, how do you think such directory structure would interact with your exdir?

[bendichter]

@SylvainTakerkart NWB files have internal structure that accommodates raw data, derived data, and metadata all in one file. The original DANDI layout deferred to NWB files for within-session data organization, and the file organization only handles the super-session info: dandiset and subject. That said, we have been talking on the DANDI team about separating raw from derived data as you have here, and we can discuss changing our format to match yours in that aspect.

[satra]

@SylvainTakerkart - thanks for the ping. here are a few thoughts on the issue.

in bids i made a suggestion for consideration of individual level data vs data that is generated from aggregation of information across individuals (bids-standard/bids-2-devel#43). it makes for a cleaner and objective separation of information that is contained in the folder. with the growing hardware ecosystem in neurophys for free behaving experiments (openephys + bonsai), we are also likely to see data that involves multiple interacting participants. we would still be able to organize information from the point of view of a participant, but that may include data about other participants.

the words rawdata, derivatives, and metadata are often linked quite directly to acquisition instruments, experimental techniques, pipelines, and implementation strategies. what is a derivative today could become raw tomorrow, as instruments integrate more processing into them. i would suggest avoiding that nomenclature if possible. this was also another reason for the participant level organization.

re: flattening in DANDI:
the primary reason is the use of NWB, which itself is an organized store of information and metadata and contains most of the details within one construct. we did not want to replicate it outside and limit any information replication to the kinds of information that a neurophys researcher may typically want from perusing the filename.

another consideration in DANDI is to consider a world of objects where the data does not hit the filesystem ever or the filesystem is an object store. this is getting increasingly true for larger data with APIs providing access to information. In such a mode the organization in a typical folder level may be a short term consideration for people working on laptops/desktops with storage that is local and in more traditional HPC settings. given some of the data coming into DANDI, data transfer in general to do things would be too expensive (from a time and perhaps from a cost perspective). therefore accessing pieces of information as necessary for computation is where we are likely headed. this doesn't mean you cannot store information in a filesystem, but we are going to be moving a little closer even to our API for our data search clients and use datalad as the filesystem model. if you consider pybids, the first thing it does is generate a database index. but unlike bids where most datasets are a few GB at most, neurophys data is being generated at much larger sizes easily hitting TBs in many situations.

our current thinking in dandi is driven to support a changing landscape in neurophysiology over the next 5 - 10 years a bit more than what people have been doing traditionally. we have been asked multiple times to store hundreds of TBs of data for single datasets. that's a scale at which most people will not be looking at filesystem organization. we can always provide a view of the underlying data through a metadata remapping model. i personally consider bids to be a practical and efficient view of a more complex information model (one that we are capturing in a more structured manner in NIDM for example).

[yarikoptic]

FWIW on the aspect of

sub-GUID/ses-YYYYMMDD_XXX_BBBB/rawdata

which exemplifies the suggestion to have filename itself without reflection of metadata encoded in the upper directories names.

I originally also was arguing for BIDS to not duplicate sub- and ses- in the filename since they are already present in the directory names. Main argumentation IIRC toward keeping them in the filename as well was: make file names unambiguous across subjects/sessions so they could be copied/shared (with collaborators) etc without causing confusion. Moreover, pragmatically, many tools display only the filename, not the full path to the file, in their listing of the files etc. So when I load multiple files in some viewer, I can tell from the filename alone to which subject they belong etc, instead of wondering from what that "rawdata" among 10 came from. In summary: I found it a bit more cumbersome but useful to have sub-, ses- in the filenames as well.