Cannot load Zarr store from S3 with square brackets (i.e., `[` and `]`) in the name

[aliddell]

Zarr version

v2.18.3

Numcodecs version

v0.11.0 (Windows), v0.13.1 (Linux)

Python Version

3.10.11 (Windows), 3.11.5 (Linux)

Operating System

Windows and Linux

Installation

pip into conda env (Windows), pip into venv (Linux)

Description

See this message on the Zulip.

I am unable to load a Zarr dataset backed by s3.S3Map with square brackets in the name. Example tested is test_stream_data_to_s3[version0-None].zarr.

It appears to be unable to find .zgroup:

raceback (most recent call last):
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\storage.py", line 2462, in __getitem__
    value = self._values_cache[key]
KeyError: '.zgroup'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\mapping.py", line 143, in __getitem__
    result = self.fs.cat(k)
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 115, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 100, in sync
    raise return_result
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 55, in _runner
    result[0] = await coro
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 405, in _cat
    paths = await self._expand_path(path, recursive=recursive)
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 758, in _expand_path
    out = await self._expand_path([path], recursive, maxdepth)
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\asyn.py", line 782, in _expand_path
    raise FileNotFoundError(path)
FileNotFoundError: ['zarr-test/test_stream_data_to_s3[version0-None].zarr/.zgroup']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\hierarchy.py", line 164, in __init__
    meta_bytes = store[mkey]
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\storage.py", line 2470, in __getitem__
    value = self._store[key]
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\storage.py", line 738, in __getitem__
    return self._mutable_mapping[key]
  File "C:\Users\Alan Liddell\.conda\envs\napari-env\lib\site-packages\fsspec\mapping.py", line 147, in __getitem__
    raise KeyError(key)
KeyError: '.zgroup'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\testing\s3-problems\hist.py", line 37, in <module>
    s3_group = get_group_from_store(get_s3_store())
  File "c:\testing\s3-problems\hist.py", line 31, in get_group_from_store
    return zarr.group(store=cache)
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\hierarchy.py", line 1358, in group
    return Group(store, read_only=False, chunk_store=chunk_store,
  File "C:\Users\Alan Liddell\AppData\Roaming\Python\Python39\site-packages\zarr\hierarchy.py", line 167, in __init__
    raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''

But we can confirm with s3fs that it's there:

import os, dotenv, s3fs

dotenv.load_dotenv()

endpoint = os.environ["ZARR_S3_ENDPOINT"]
bucket = os.environ["ZARR_S3_BUCKET_NAME"]
key_id = os.environ["ZARR_S3_ACCESS_KEY_ID"]
key_secret = os.environ["ZARR_S3_SECRET_ACCESS_KEY"]

store_path = "test_stream_data_to_s3[version0-None].zarr"

s3 = s3fs.S3FileSystem(
    key=key_id, secret=key_secret, client_kwargs={"endpoint_url": endpoint}
)
s3.ls(f"{bucket}/{store_path}/.zgroup")

Steps to reproduce

import os, dotenv, s3fs, zarr

dotenv.load_dotenv()

endpoint = os.environ["ZARR_S3_ENDPOINT"]
bucket = os.environ["ZARR_S3_BUCKET_NAME"]
key_id = os.environ["ZARR_S3_ACCESS_KEY_ID"]
key_secret = os.environ["ZARR_S3_SECRET_ACCESS_KEY"]

store_path = "test_stream_data_to_s3[version0-None].zarr"

s3 = s3fs.S3FileSystem(
    key=key_id, secret=key_secret, client_kwargs={"endpoint_url": endpoint}
)
# print(s3.ls(f"{bucket}/{store_path}/.zgroup")) # uncomment for a sanity check

store = s3fs.S3Map(root=f"{bucket}/{store_path}", s3=s3)
group = zarr.group(store=store)

Additional output

No response

[d-v-b]

have you tried zarr-python 3.x with the same data?

[aliddell]

@d-v-b

I've tried, but it's not clear from the docs how to do it.

# ...
store = s3fs.S3Map(root=f"{bucket}/{store_path}", s3=s3)
remote_store = zarr.storage.RemoteStore(store)

throws

AttributeError                            Traceback (most recent call last)
Cell In[10], line 2
      1 store = s3fs.S3Map(root=f"{bucket}/{store_path}", s3=s3)
----> 2 remote_store = zarr.storage.RemoteStore(store)

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/storage/remote.py:72, in RemoteStore.__init__(self, fs, mode, path, allowed_exceptions)
     69 self.path = path
     70 self.allowed_exceptions = allowed_exceptions
---> 72 if not self.fs.async_impl:
     73     raise TypeError("Filesystem needs to support async operations.")

AttributeError: 'FSMap' object has no attribute 'async_impl'

The example on the tutorial page here ("latest" docs for version 3.0.0b2.dev18+g4c3081c) is wrong. I get to the 4th line and:

In [12]: root = zarr.group(store=store)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 root = zarr.group(store=store)

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/_compat.py:43, in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     41 extra_args = len(args) - len(all_args)
     42 if extra_args <= 0:
---> 43     return f(*args, **kwargs)
     45 # extra_args > 0
     46 args_msg = [
     47     f"{name}={arg}"
     48     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:], strict=False)
     49 ]

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/api/synchronous.py:182, in group(store, overwrite, chunk_store, cache_attrs, synchronizer, path, zarr_version, zarr_format, meta_array, attributes)
    167 @_deprecate_positional_args
    168 def group(
    169     store: StoreLike | None = None,
   (...)
    179     attributes: dict[str, JSON] | None = None,
    180 ) -> Group:
    181     return Group(
--> 182         sync(
    183             async_api.group(
    184                 store=store,
    185                 overwrite=overwrite,
    186                 chunk_store=chunk_store,
    187                 cache_attrs=cache_attrs,
    188                 synchronizer=synchronizer,
    189                 path=path,
    190                 zarr_version=zarr_version,
    191                 zarr_format=zarr_format,
    192                 meta_array=meta_array,
    193                 attributes=attributes,
    194             )
    195         )
    196     )

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/core/sync.py:141, in sync(coro, loop, timeout)
    138 return_result = next(iter(finished)).result()
    140 if isinstance(return_result, BaseException):
--> 141     raise return_result
    142 else:
    143     return return_result

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/core/sync.py:100, in _runner(coro)
     95 """
     96 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
     97 exception, the exception will be returned.
     98 """
     99 try:
--> 100     return await coro
    101 except Exception as ex:
    102     return ex

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/api/asynchronous.py:577, in group(store, overwrite, chunk_store, cache_attrs, synchronizer, path, zarr_version, zarr_format, meta_array, attributes, storage_options)
    573 zarr_format = _handle_zarr_version_or_format(zarr_version=zarr_version, zarr_format=zarr_format)
    575 mode = None if isinstance(store, Store) else cast(AccessModeLiteral, "a")
--> 577 store_path = await make_store_path(store, path=path, mode=mode, storage_options=storage_options)
    579 if chunk_store is not None:
    580     warnings.warn("chunk_store is not yet implemented", RuntimeWarning, stacklevel=2)

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/storage/common.py:257, in make_store_path(store_like, path, mode, storage_options)
    255 else:
    256     msg = f"Unsupported type for store_like: '{type(store_like).__name__}'"  # type: ignore[unreachable]
--> 257     raise TypeError(msg)
    259 if storage_options and not used_storage_options:
    260     msg = "'storage_options' was provided but unused. 'storage_options' is only used for fsspec filesystem stores."

TypeError: Unsupported type for store_like: 'FSMap'

I also tried

store = zarr.storage.RemoteStore.from_url(
    f"{bucket}/{store_path}",
    storage_options={
        "key": key_id,
        "secret": key_secret,
        "client_kwargs": {"endpoint_url": endpoint},
    },
)

and I get

TypeError                                 Traceback (most recent call last)
Cell In[24], line 1
----> 1 store = zarr.storage.RemoteStore.from_url(
      2     f"{bucket}/{store_path}",
      3     storage_options={
      4         "key": key_id,
      5         "secret": key_secret,
      6         "client_kwargs": {"endpoint_url": endpoint},
      7     },
      8 )

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/storage/remote.py:134, in RemoteStore.from_url(cls, url, storage_options, mode, allowed_exceptions)
    114 """
    115 Create a RemoteStore from a URL.
    116 
   (...)
    131 RemoteStore
    132 """
    133 fs, path = fsspec.url_to_fs(url, **storage_options)
--> 134 return cls(fs=fs, path=path, mode=mode, allowed_exceptions=allowed_exceptions)

File ~/repos/acquire-zarr/venv/lib/python3.11/site-packages/zarr/storage/remote.py:73, in RemoteStore.__init__(self, fs, mode, path, allowed_exceptions)
     70 self.allowed_exceptions = allowed_exceptions
     72 if not self.fs.async_impl:
---> 73     raise TypeError("Filesystem needs to support async operations.")

TypeError: Filesystem needs to support async operations.

Passing "asynchronous": True in storage_options doesn't seem to make a difference.

Any ideas?

[d-v-b]

@aliddell our docs should definitely improve here! I think you should be able to just run this:

obj = zarr.open("s3://bucket/path", storage_options=storage_options)

store = zarr.storage.RemoteStore.from_url(
f"{bucket}/{store_path}",
storage_options={
"key": key_id,
"secret": key_secret,
"client_kwargs": {"endpoint_url": endpoint},
},
)

this was close, but the s3 URL for a s3 bucket starts with s3://, so I think this would also work:

store = zarr.storage.RemoteStore.from_url(
    f"s3://{bucket}/{store_path}",
    storage_options={
        "key": key_id,
        "secret": key_secret,
        "client_kwargs": {"endpoint_url": endpoint},
    },
)

[aliddell]

@d-v-b

I needed to specify mode="r+", but that seems to have done it.

In [20]: store = zarr.storage.RemoteStore.from_url(
    ...:     f"s3://{bucket}/{store_path}",
    ...:     storage_options={
    ...:         "key": key_id,
    ...:         "secret": key_secret,
    ...:         "client_kwargs": {"endpoint_url": endpoint},
    ...:     },
    ...:     mode="r+",
    ...: )
    ...: group = zarr.group(store=store)

In [21]: group
Out[21]: <Group <RemoteStore(S3FileSystem, zarr-test/test_stream_data_to_s3[version0-None].zarr)>>

[d-v-b]

glad to hear that it worked, sorry the docs are opaque, and for the mode thing we should probably expose read, read_array, read_group functions that don't take a mode argument

[aliddell]

Already fixed then. I'll close.