Reading and writing scalar arrays

[TomNicholas]

I was trying to change the values of all the arrays in a zarr store in-place (to alter their dtype), but kept getting tripped up when one of the arrays was a scalar.

As far as I can tell from the documentation the recommended way to read a zarr array into an in-memory numpy array is to slice it, i.e.

numpy_array = store['key'][:]

But this will raise an indexing error if a scalar is stored at store['key']!

IndexError: too many indices for array; expected 0, got 1

I managed to load the scalar by doing

numpy_array = np.array(store['key'])

but when I later want to overwrite the scalar in the store with some new values, I hit the same indexing error (and the np.array constructor trick can't help me here)

store['key'][:] = numpy_array

This shouldn't happen - scalars are special cases of arrays, so anything that works for an N-D array should also work for a 0-D array.

Perhaps there is another method that I've missed, but then it should be linked to in the Reading and Writing Data section of the tutorial page.

I think this also relates to the discussion in #1603 - this problem only comes up because the [:] syntax for loading is itself a bit of an abuse of how slicing is supposed to work (see the argument about typing in that thread).

EDIT: Apparently my use case doesn't have a simple solution 😞 #1444

[d-v-b]

to achieve the goal of changing the array in-place, you can use the array.store API to directly modify every chunk of the array as you see fit, and once you've done that you can modify the array metadata through the same API. clients reading from your array will get errors during this process, but in the end it should work

[d-v-b]

I'm broadly interested in the world of changing arrays in-place -- I think an important ingredient here is something that durably tracks the progress of the transformation, so that your array doesn't get irreversibly corrupted if your job fails. if anyone knows any existing tools for this, please share them!

[rabernat]

I'm not sure Tom is particularly concerned about transactions or safety in his question. It's more about Zarr's handling of scalar values.

However, at the risk of going off topic, the question of safely updating Zarr arrays is one of the main problems we have tackled with Arraylake -- https://docs.earthmover.io/concepts/version_control_system -- however, we chose to tackle it by building a customized store which supports transactions, rather than by making any changes to Zarr itself.

[d-v-b]

It's more about Zarr's handling of scalar values.

Right, on this topic I wonder if there's something missing from our test suite? I thought the 0D case was tested pretty extensively, but apparently not extensively enough

[TomNicholas]

Yes I was just trying to point out an issue with the [:] syntax for scalars.

I think it would be cool if zarr-python had some simplified support for "try to update this, if it fails then rollback", and I was about to suggest something, but after reading that arraylake docs page I think that was probably naive! 😅

[rabernat]

The one thing that could easily be done is to stage the changes in memory and then commit them all at once. This is how tensorstore transactions work. That's a fine solution for small updates that can fit in memory.