Unwritten Zarr v3 arrays values should default to NaN

[shoyer]

What happened?

With Zarr v2, Xarray conflated _FillValue and fill_value (#5475), so unwritten data in a Zarr file is always decoded as NaN.

In Zarr v3, this changed. Now, it appears that unwritten data in a Zarr file uses the default Zarr fill_value=0.

In practice, this means that is Zarr metadata is written to disk (with compute=False) but no values are written, the data will be decoded by xarray as all zeros instead of all NaNs. This is perhaps unavoidable for integer data (there is no integer NaN), but for floats, we should default to a user controllable fill_value of NaN, which is clearly an invalid value.

To reproduce:

import xarray
import numpy as np

ds = xarray.Dataset({'foo': ('x', np.ones(3))}).chunk()

path = '/tmp/foo5.zarr'
ds.to_zarr(path, compute=False)

# all NaN with zarr v2, all zeros with zarr v3
print(xarray.open_zarr(path).compute())

Possibly related: #10633, #10269

[d70-t]

I don't have solutions but maybe some opinion:

Essentially, ds.to_zarr(path, compute=False) in my mind would mean: "just write the metadata, dont' care about the content's I'll work on it later, trust me". It implicitly means "use the fill_value (or I sometimes like to say background value)" to fill the data space. But it doesn't mean "flood the dataset with missing value indicators".

I could see quite valid use cases for this, e.g. I want to create a dataset which covers a large space and contains counts of some events, or maybe differentes which uses might want to collect over time and then use later on. In both cases, quickly initializing a huge array with 0 values all over is a valuable thing, and exactly what I'd expect ds.to_zarr(path, compute=False) to do.

But I'd also think, that for data types which contain an indicator for missing data (e.g. float contains NaN), using this indicator as the "background value" is the most reasonable thing to do. In this case, we still wouldn't have a "missing value" (in the sense of CF-Encoding), but still our background values end up being NaN, because they are NaN.
For integers (which don't contain any missing value indicators) this would essentially behave the same way (we don't have a "missing value") the only thing which is different is that the background value is a bit more random and thus probably more surprising to some.

TL;DR: I think 0.0 is just a very bad choice for a default fill_value for float, but 0 might be ok for int.

[shoyer]

Essentially, ds.to_zarr(path, compute=False) in my mind would mean: "just write the metadata, dont' care about the content's I'll work on it later, trust me". It implicitly means "use the fill_value (or I sometimes like to say background value)" to fill the data space. But it doesn't mean "flood the dataset with missing value indicators".

👍

If most cases, to_zarr(path, compute=False) really means to_zarr(path, metadata_only=True): #8343

TL;DR: I think 0.0 is just a very bad choice for a default fill_value for float, but 0 might be ok for int.

Agreed! So maybe the solution is just to change the default fill_value for to_zarr() to be NaN for float dtypes, and to clearly document how to change that (e.g., via encoding, or maybe a simpler argument zarr_fill_value that holds for all variables).

[shoyer]

Agreed! So maybe the solution is just to change the default fill_value for to_zarr() to be NaN for float dtypes, and to clearly document how to change that (e.g., via encoding, or maybe a simpler argument zarr_fill_value that holds for all variables).

One thing that makes this particularly tricky right now is that on Zarr v3 , you would set something like encoding=dict.fromkey(ds.data_vars, {'fill_value': 'nan'}) but with Zarr v2 this raises an error from Xarray:

ValueError: unexpected encoding parameters for zarr backend:  ['fill_value']. Use `_FillValue` to set the Zarr array `fill_value`

[d70-t]

I agree, this is annoying.

My (likely biased and possibly impractical) opinion on this would be, that treating fill_value and _FillValue as the same thing is a defect of zarr v2 (and maybe even all the way back, one of the netCDF NUG). I think the proper way of handling this defect would be to fix it in all things down the line.
For Zarr v2, this would likely mean to start differentiating between the two and write both (as outlined in #5475 (comment)), to eventually allow having the two with different values and not implying _FillValue because of the presence of fill_value. This would also mean, specifying fill_value in encoding without _FillValue should be legal, even in v2, and even in netCDF.

But yes, this would probably imply that some existing datasets would have to be interpreted differently in future, which might be a prohibitive issue.

[rabernat]

Xarray could make the choice to set the default Fill Value to NaN for V3 float arrays. That doesn't require any changes from Zarr. I would support that.

[shoyer]

Xarray could make the choice to set the default Fill Value to NaN for V3 float arrays. That doesn't require any changes from Zarr. I would support that.

See #10757 for my proposed fix along these lines.