How to prevent Zarr from returning NaN for missing chunks?

[willirath]

Is there a way of preventing Zarr from returning NaNs if a chunk is missing?

Background of my question: We're seeing problems with either copying data to GCS or with GCS having problems to reliably serve all chunks of a Zarr store.

In arr below, there's two types of NaN filled chunks returned by Zarr.

from dask import array as darr
import numpy as np

arr = darr.from_zarr(""gs://pangeo-data/eNATL60-BLBT02X-ssh/sossheig/")

First, there's a chunk that is completely flagged missing in the data (chunk is over land in an Ocean dataset) but present on GCS (https://console.cloud.google.com/storage/browser/_details/pangeo-data/eNATL60-BLBT02X-ssh/sossheig/0.0.0) and Zarr correctly find all items marked as invalid:

np.isnan(arr.blocks[0, 0, 0]).mean().compute()
# -> 1.0

Then, there's a chunk (https://console.cloud.google.com/storage/browser/_details/pangeo-data/eNATL60-BLBT02X-ssh/sossheig/0.7.3) that is not present (at the time of writing this, I get a "load failed" and a tracking id from GCS) and Zarr returns all items marked invalid as well:

np.isnan(arr.blocks[0, 7, 3]).mean().compute()
# -> 1.0

How do I make Zarr raise an Exception on the latter?

cc: @auraoupa
related: pangeo-data/pangeo#691

[willirath]

Looks like it's necessary to override

zarr-python/zarr/core.py

Line 1581 in e7708c9

except KeyError:

et al to make zarr raise on non-existing chunks?

[jakirkham]

Have you tried setting the fill value?

[alimanfoo]

Yes this is not currently supported, but would be straightforward to add. I've actually been thinking myself recently that this would be useful for very similar reasons, i.e., when you want to ensure that an exception is raised when trying to access data from a missing chunk. One way to achieve this could be to add a mechanism for activating this behaviour explicitly. E.g., something like: z = ... # some zarr array z.set_options(fill_missing_chunk=False) Thoughts and suggestions welcome.

…

On Sat, 19 Oct 2019, 15:34 Willi Rath, ***@***.***> wrote: Looks like it's necessary to override https://github.com/zarr-developers/zarr-python/blob/e7708c948d2c0ff91863d675c1072b0dfe9ce2a6/zarr/core.py#L1581 et al to make zarr raise on non-existing chunks? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#486?email_source=notifications&email_token=AAFLYQXHM2WFRK5BDATMIDDQPMLIDA5CNFSM4JCPOTIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBXRXRA#issuecomment-544152516>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFLYQXAROADSJEALO3X4YLQPMLIDANCNFSM4JCPOTIA> .

[alimanfoo]

Just to say that I still think this is a valid thing to address in zarr, but there could/should also be some work upstream in fsspec to ensure that KeyError is only ever raised by the store in the case where it is certain that the underlying file/object does not exist. If some other problem is encountered such as a transient error attempting to read the file/object, then fsspec should raise as some other kind of error, which would get propagated up through zarr. I've raised an issue here: fsspec/filesystem_spec#255

[jakirkham]

cc @martindurant

[martindurant]

(was fixed in fsspec/filesystem_spec#259 )

[alimanfoo]

Thanks @martindurant for the upstream fix.

I think I will reopen this issue, however, as there may still be use cases where you want to change zarr's behaviour. E.g., you may know that you definitely do have some missing chunks in the data, and you want to make sure you don't accidentally request any data from a region overlapping a missing chunk.

[delgadom]

Hi @alimanfoo and @willirath - thanks for raising this. Just ran across this issue while using zarr on google cloud. With huge jobs there are always I/O failures, but I never would have expected this behavior:

import xarray as xr
import numpy as np
import os

ds = xr.Dataset({
    'x': [0, 1],
    'y': [0, 1],
    'myarr': (('x', 'y'), [[0., np.nan], [2., 3.]]),
})

ds.chunk({'x': 1, 'y': 1}).to_zarr('myzarr.zarr')

# chunk file disappears due to bad write, storage failure, gremlins...
os.remove('myzarr.zarr/myarr/1.0')

# I would LOVE for this to return an error
read_ds = xr.open_zarr('myzarr.zarr').compute()

# instead the error is here
xr.testing.assert_equal(ds, read_ds)

This seems pretty problematic when NaNs are actually meaningful and without doing additional inspection of the filesystem and zarr metadata it's impossible to know if a NaN is a NaN or a failed write from a previous step, or just the chaos monkeys that live on cloud systems generally.

Is there a patch you can recommend as a workaround? I'm less familiar with the zarr API as I've only used it via xarray.

[joshmoore]

Did you see #489 (comment), @delgadom? Perhaps give that some testing to help drive it forward?

[martindurant]

The fspsec mapper is designed to be able to specify which basic exceptions get turned into KeyError (i.e., missing data) and which do not. I cannot quite seem to make ti work for this use-case though, I expect the code needs a little work. Also, there is a difference between creating an fsspec mapper instance and passing it to open_zarr versus passing a "file://..." URL and opening it in zarr (which uses FSStore, a wrapper with its own exception catching mechanism).

I think the bug is in zarr _chunk_getitems calling .getitems(ckeys, on_error="omit") - it should raise for any error except KeyError.

[joshmoore]

@martindurant : do you assume #489 is unneeded then?