v3 core spec: Consider to drop /meta prefix, have file at URI

[jstriebel]

Citing @jbms from #149 (comment):

As discussed on the community meeting, this naming scheme has the drawback in that there is not a good way to have a path directly to a non-root array. Additionally, it was noted that to better integrate with existing filesystem completion in editors, etc., it would be helpful if the path were a real filesystem path.

A few proposals were made:

• Instead of "meta/root" + P + ".group.json" as the key for the metadata file, instead have it just be: P + ".zr3". This would lead to a key of just ".zr3" for the root. It was noted though that with a consolidated metadata extension these metadata files would not actually exist. The data could be placed in e.g. "_data" or something, where names beginning with an underscore could be reserved to prevent conflicts. This would still require a way to locate the root directory --- that could be done by storing the relative path inside the array metadata.
• Alternatively, we could leave the directory structure alone but require an extension as part of the directory containing the root, e.g. "foo.zr3", and disallow any array or group names from ending in ".zr3". Then you could use: "path/to/root.zr3/path/to/array" as a pseudo-path to an individual array. The downside is that it may be confusing to use something that looks like a path, and where a portion corresponds to a real filesystem path but a portion does not. File completion in editors also wouldn't support this.
• We could use a special syntax to combine both a path to a root and a path to an array into a single string, e.g. "path/to/root//path/to/array" or "path/to/root#/path/to/array". We would need to carefully choose the syntax to avoid conflicts with e.g. fsspec, and file completion in editors also wouldn't support this.

Some more comments from discussion rounds I remember:

• .json suffix is useful to have a correct mimetype by default for many stores, e.g. S3
• Having a URI (TBD, see v3: Define standard "URL" syntax for referencing a specific array, group, attribute within a zarr repository #132) to an array or group, it would be great to actually find a directory or file there. E.g. s3://bucket-name/key-name/name-of-the-zarr-path.zarr/hierarchy/path/my-data.array.json could be a URI to point to the my-data array at the path hierarchy/path/my-data of the zarr hierarchy which is placed under s3://bucket-name/key-name/name-of-the-zarr-path.zarr/. (Just made up a URI here as an example, feel free to discuss this in v3: Define standard "URL" syntax for referencing a specific array, group, attribute within a zarr repository #132). Using such a URI schema and dropping the /meta prefix, one could find the relevant file (at least for filesystem or http stores or using appropriate clients for other stores).
• The original motivation to have /meta and /data separate is to be able to list all meta keys without also listing the chunk files for efficiency reasons. If it's possible to exclude directories for key-listings for most relevant stores, only using a prefix for the chunk files would still give this efficiency, but it's unclear if that's the case.
• It might be useful to be able to place chunk-files in arbitrary locations (possibly even other stores). This could be added as an extension, but can also be considered for the core spec.

Pinging discussion participants I remember so far: @joshmoore @jbms @rabernat @WardF

[jbms]

Due to the global storage transformers proposal #182 from @rabernat, my opinion on this has changed a bit:

• With a global storage transformer in use, there is no guarantee that the array metadata exists as a file in the underlying store. Therefore, the URL syntax should not depend on that.
• Having something that looks like a file path, "path_to_zarr.zarr/group/array", but where "path_to_zarr.zarr" is real directory, but "group/array" may not be contained as a file/directory within it, may be confusing and lead users unfamiliar with the special convention to believe files are missing. Instead, I think it would be better to use a more explicit syntax, like "path_to_zarr.zarr#/path/to/array" since that avoids any confusion about which part is a real path, and will work even if users do not use the suggested ".zarr" file extension.
• The "meta" and "data" prefixes were intended to allow efficiently listing all of the metadata files when using stores like s3, gcs, and other "pure" key-value stores that do not have real directories. However, it provides no benefit for filesystem-backed stores that do have directories. Furthermore: both s3 and gcs support for emulating directories via the "delimiter" option to the list API, which allows efficient directory-like listing. Therefore, it is not clear that this naming scheme is really needed for the most common use case for which it seems to have been designed.
• This naming scheme still doesn't work that well if you have a very large number of arrays --- for example if you have 1 million arrays within a hierarchy of groups (not implausible for some use cases), if the underlying store doesn't support directory-based listing, then to list all of the top-level groups would require listing all million arrays. There are alternative naming schemes that would add directory support on top of a pure key-value store, that could be implemented as a storage transformer. For example, each directory is assigned a uuid (similar to how directories are implemented in filesystems).
• A natural extension of global storage transformers is per-group storage transformers --- this could be used to link an entire group from some external location. Per-group storage transformers are much more naturally defined if each group corresponds to a single prefix in the key space of the underlying store.

In summary, I think zarr v3 should just use a zarr v2-like naming scheme by default, and rely on a storage transformer extension to provide an alternative naming scheme if it is desired.

This would also allow the root metadata file, and the concept of a root, to be eliminated entirely --- instead just the group metadata file could be used.

[rabernat]

There's a lot to unpack here, but let me just respond to one specific point first.

• The "meta" and "data" prefixes were intended to allow efficiently listing all of the metadata files when using stores like s3, gcs, and other "pure" key-value stores that do not have real directories. However, it provides no benefit for filesystem-backed stores that do have directories.

Even on traditional POSIX filesystems, there is a strong benefit to separating metadata and chunks into separate directories. Here's an example: we frequently use Zarr on NASA's Pleiades supercomputer. This system has a large shared filesystem which performs well for some operations. However, listing directories with a large number of files in them can be extremely slow. Separating metadata from chunks allows us to quickly discover the structure of a Zarr hierarchy without having to list millions of chunks.

[jstriebel]

Interesting idea, I guess that's also where the motivation for #184 is coming from, @jbms?

• With a global storage transformer in use, there is no guarantee that the array metadata exists as a file in the underlying store. Therefore, the URL syntax should not depend on that.

True, especially with something like consolidated metadata.

• Having something that looks like a file path, "path_to_zarr.zarr/group/array", but where "path_to_zarr.zarr" is real directory, but "group/array" may not be contained as a file/directory within it, may be confusing and lead users unfamiliar with the special convention to believe files are missing. Instead, I think it would be better to use a more explicit syntax, like "path_to_zarr.zarr#/path/to/array" since that avoids any confusion about which part is a real path, and will work even if users do not use the suggested ".zarr" file extension.

There's a similar discussion going on for OME-NGFF: 144

[jbms]

There's a lot to unpack here, but let me just respond to one specific point first.

• The "meta" and "data" prefixes were intended to allow efficiently listing all of the metadata files when using stores like s3, gcs, and other "pure" key-value stores that do not have real directories. However, it provides no benefit for filesystem-backed stores that do have directories.

Even on traditional POSIX filesystems, there is a strong benefit to separating metadata and chunks into separate directories. Here's an example: we frequently use Zarr on NASA's Pleiades supercomputer. This system has a large shared filesystem which performs well for some operations. However, listing directories with a large number of files in them can be extremely slow. Separating metadata from chunks allows us to quickly discover the structure of a Zarr hierarchy without having to list millions of chunks.

While it might be tricky to accomplish with regular shell utilities like GNU find, I think it can be done from zarr implementations relatively easily: start at the root, check if the current directory is an array based on presence of array metadata, if not, list it and recurse on subdirectories.

[rabernat]

• With a global storage transformer in use, there is no guarantee that the array metadata exists as a file in the underlying store. Therefore, the URL syntax should not depend on that.

I agree with this.

• Instead, I think it would be better to use a more explicit syntax, like "path_to_zarr.zarr#/path/to/array" since that avoids any confusion about which part is a real path, and will work even if users do not use the suggested ".zarr" file extension.

I like this idea in general. However, I have found it to be problematic that the entry point to a zarr store is a directory, not an actual file, because in some contexts (e.g. S3) it is impossible to tell whether a directory exists or not. Given that the first thing that all V3 stores must do is open the root metadata document, wouldn't it make more sense to use that as the actual file / URI identifying the zarr store, e.g.

protocol://path/to/store/zarr.json#/path/to/array

This opens the question of whether the root group actually needs to be named zarr.json or could have any name.

• Therefore, it is not clear that this naming scheme is really needed for the most common use case for which it seems to have been designed.

This is an interesting suggestion which I think we should examine closely in today's call. It would be particularly helpful if anyone could recall the original use cases that motivated the new layout proposal. I am trying to think of one (e.g. a case where you are forced to list a directory with millions of chunks in it just to find a metadata document), but I am not coming up with anything. 🤔

• This naming scheme still doesn't work that well if you have a very large number of arrays --- for example if you have 1 million arrays within a hierarchy of groups (not implausible for some use cases), if the underlying store doesn't support directory-based listing, then to list all of the top-level groups would require listing all million arrays.

What is this hypothetical store that doesn't support directory-based listing? As you mentioned above, it's not actually S3 / GCS, since they effectively support directory listing. So is it even an important case to consider? In your previous point, you used the fact that S3 does support directory listing as an argument for reverting to the unified storage layout. If your million arrays are sufficiently nested, traversing the directory tree should be feasible, no?

• A natural extension of global storage transformers is per-group storage transformers --- this could be used to link an entire group from some external location. Per-group storage transformers are much more naturally defined if each group corresponds to a single prefix in the key space of the underlying store.

This makes sense to me. Basically a generalization of the global storage transformers idea (#182) to act at the group level.

---

Perhaps one path forward would be to go back to the unified storage layout as the default, and then redefine the separate data / meta layout as a global (or possible group-level) storage transformer (here called storage_layout).

{
  "storage_transformers": [
    {
      "extension": "https://purl.org/zarr/spec/storage_transformers/storage_layout/1.0",
      "configuration": {
        "metadata_path": "./meta/root/",
        "chunk_path": "./data/root/"
      }
    }
  ]
}

This could be used to provide all sorts of redirection to different storage locations, e.g. pointing at a completely different service. (I would actual prefer this over being able to specify a seperate chunkstore object at runtime.)

[jstriebel]

Crosslinking the discussion from the ZEP meeting yesterday:
https://hackmd.io/ZilORe8AQvyqH6ArqDw0Cg?view

Two action items regarding this issue:

• @rabernat to do some performance benchmarking
• @jbms to propose a new storage layout for v3

The agreement seemed to be to have somthing similar to the v2 format, but with an extra folder for the chunk-data per array. For an array foo in group mygroup it could look similar to:

mygroup/foo/array.json
mygroup/foo/chunks/0/0
mygroup/foo/chunks/0/1
…

[jbms]

Before I create a PR, I wanted to discuss a few options for the new storage layout:

The basic assumptions I'm making for the default storage layout are:

• No special root. The root is just a group that happens to not have a parent group.
• Arrays can be standalone.
• Arrays and groups are each fully contained within a single prefix/directory.

In my mind, the key question to consider is whether the directory name should encode whether a group member is an array, a group, or perhaps something else, like a bare key-value store prefix to be used by some extension, such as for storing meshes.

---

If don't encode the group member type in the name (as in zarr v2), then we have a layout like:

Option 1:

(Bare array)
zarr.json
0/0/0

(Group)
zarr.json
subgroup/zarr.json
subgroup/myarray/zarr.json
subgroup/myarray/0/0/0
myarray/zarr.json
myarray/0/0/0

Pros:

• Simplest mapping of zarr logical namespace to the underlying store
• Prevents a group and array having the same name, in a race-free way assuming the underlying store supports atomic operations on a single key.
• A query to determine if a given member is present, "myarray" in group, efficiently maps to an existence check on the underlying store.

Cons:

• When listing a group's members, determining whether a group member is a subgroup or an array requires one extra operation per member (reading the zarr.json key). For a large group (e.g. 1 million members), this operation becomes extremely expensive.

---

If we encode the group member type in the name, then we have:

Option 2:

(Bare array)
zarr.json
0/0/0

(Group)
subgroup.zgroup/zarr.json
subgroup.zgroup/myarray.zarray/zarr.json
subgroup.zgroup/myarray.zarray/0/0/0
myarray.zarray/zarr.json
myarray.zarray/0/0/0

Option 2A: (use a file extension only for arrays, not for subgroups, or vice versa)

(Bare array)
zarr.json
0/0/0

(Group)
subgroup/zarr.json
subgroup/myarray.zarray/zarr.json
subgroup/myarray.zarray/0/0/0
myarray.zarray/zarr.json
myarray.zarray/0/0/0

Pros:

• When listing a group's members, simply listing the group's "directory" is sufficient to determine the type of each group member.

Cons:

• The added extensions add extra "noise" in the keys used in the underlying store.
• With Option 2A, we need to impose additional requirements on group member names (namely that they cannot end in ".zarray") to avoid ambiguity.
• Preventing the creation of a subgroup and an array with the same name requires an extra check on every array creation within a group (which is not an insignificant cost), and is racy if the underlying store does not support multi-key transactions (most stores in current use do not support such transactions.
• Checking if a given group member is present, if we don't know the type of group member, may require multiple queries on the underlying store.

With this storage layout, there are two possible solutions to the issue of avoiding having a group and array with the same name:
i. Expose the file extensions to the user as part of the group member name. For example, it would be an error for a user to attempt to create/open a "myarray" array within a group; instead, they would need to create a "myarray.zarray" array. It would be allowed to have a "myarray" subgroup and a "myarray.array" array within the same group. These special naming requirements may create challenges for code intended to work with multiple hierarchical storage formats, e.g. zarr v2, zarr v3, n5, hdf5.
ii. Leave the prevention of a group and array with the same name up to users; we can require in the spec that users avoid this, but not require implementations to check for this when creating a new group or array. If a group and array with the same name are found when listing a group, the implementation could return a warning or error, but should still allow opening both the array and the group using appropriate APIs.
iii. Explicitly allow a group and array to have the same name, and merely recommend against it. APIs for listing a group would need a way to handle the duplicate names, e.g. subgroups might be returned as "subgroup/" while arrays are returned as "array".

---

It is not clear to me how important the "list group members including their type" operation is; if that operation is important, then Option 2 is better.

But currently I lean towards Option 1 since it is simpler, and Option 2 could potentially be provided via an extension.

---

Note that the chunks of an array could optionally be nested within a chunks/ prefix, e.g. for Option 1

(Bare array)
zarr.json
chunks/0/0/0

(Group)
zarr.json
subgroup/zarr.json
subgroup/myarray/zarr.json
subgroup/myarray/chunks/0/0/0
myarray/zarr.json
myarray/chunks/0/0/0

This naming choice appears to be orthogonal to the choice between Option 1 and Option 2.

I don't think it provides any benefit to a zarr implementation, but may provide a better user experience for users interactively browsing a store, e.g. using path completion --- without a "chunks" prefix, it is easy for a user to accidentally list all the chunks of an array. The "chunks" prefix would serve as a warning, and users could avoid accidentally listing the chunks.

I don't have a strong opinion on whether to use the "chunks/" prefix.

[jstriebel]

Thanks for the great write-up @jbms!

But currently I lean towards Option 1 since it is simpler, and Option 2 could potentially be provided via an extension.

+1 from me.

I don't have a strong opinion on whether to use the "chunks/" prefix.

I'd prefer to use a prefix (maybe rather _chunks, if we prohibit leading underscores in filenames, see #56 (comment)).
This could also be used to signal if an entry is a group or array by explicitly specifiying that groups must not contain a _chunks subdirectory. Then only the existance of a _chunks prefix in a directory entry is enough and arrays and groups can be distinguished via key listings.

PS: I just openened a separate issue about dropping the entrypoint metadata / explicit root: #192 I think it's fair to discuss those two things separately, but 👍 for waiting on a decision there before preparing a PR.