Bug: Using GCS or S3, FSStore doesn't pass check and create kwargs to FSSPEC's get_mapper() #775
Description
I think this is a bug, if everyone agrees I can fix it and do a PR.
Minimal, reproducible code sample, a copy-pastable example if possible
import zarr
# Works fine with this as expected
url = '~/my_root.zarr'
# Doesn't work with:
url = r'gcs://my_bucket/my_root.zarr'
# or
url = r's3://my_bucket/my_root.zarr'
storage_options = {'key': my_key, 'secret': my_secret, 'token': my_token} # Example for s3
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
zarr.open(store, 'w')Problem description
The above code fails because the FSMap created and used by FSStore did not receive the check and create kwargs (Traceback at the bottom of the post).
The check parameter is used to verify write access to remote stores.
The create function creates the root object/directory if it doesn't exist, so it can check for write access
The fsspec.mapping.get_mapper used in zarr.storage.FSStore expects explicit kwargs for check and create to send FSMap and passes storage_options aka. other kwargs to ONLY to url_to_fs (here). The FSMap call here expects the check and create kwargs, which explicitly have to be passed to get_mapper.
The issue can be circumvented by dropping the write access checking, but that may not be always the case user wants.
The ideal fix is to make zarr.storage.FSStore to expect check and create kwargs; and default to False, just like FSMap.
Version and installation information
Please provide the following:
zarr.__version__2.8.1numcodecs.__version__0.7.3- Python 3.9.5
- Linux x86_64
- installed via conda
Traceback:
Traceback (most recent call last):
File "/home/my_user/develop/my_library/benchmarks/test.py", line 5, in <module>
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/zarr/storage.py", line 1052, in __init__
self.map = fsspec.get_mapper(url, **storage_options)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 224, in get_mapper
return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)
File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 52, in __init__
raise ValueError(
ValueError: Path my_bucket/my_root.zarr does not exist. Create with the ``create=True`` keyword