Exposing chunk array types in dask.array (e.g., sparsed and masked)

[shoyer]

As noted in pydata/xarray#1769, dask currently does not expose any way to determine the computed type of a dask array containing MaskedArray or sparse blocks.

This is untenable for building complex codebases using dask: we need to know the type of subarrays, so errors can be raised at graph building instead of compute time. For masked arrays, this is merely inconvenient, but for mixing up dense/sparse arrays this is a very serious concern, because it could entail loading very large arrays into memory.

I would suggest a simple hierarchy:

• BaseArray: abstract base class for all dask array types
• Array(BaseArray): base numpy.ndarray elements
• MaskedArray(BaseArray): numpy.ma.MaskedArray elements
• SparseArray(BaseArray): sparse array elements.

[mrocklin]

Another option would be to keep a container class on the array or an example element. These have the advantage that we can avoid having to reproduce all of the logic for when various types turn into other types, which I suspect would be a significant effort to apply to the entire codebase.

Also, another issue, dask.array supports mixing container types.

[shoyer]

Another option would be to keep a container class on the array or an example element.

Sure, I would be happy with an array_type/chunk_type attribute, analogous to dtype on numpy arrays.

I'd like something that could (at least in principle) be typed checked by a system like mypy, for which both subclasses and array-types satisfy. Otherwise, it's quite likely that somebody is going to try to compute nansum() on a masked array (e.g., in xarray), with completely unknown results.

One downside of using array_type is that np.ma.MaskedArray has the design mistake of being a np.ndarray subclass. This makes it slightly awkward to distinguish masked/non-masked arrays (e.g., you need to use type() instead of isinstance()), although we do already deal with this for NumPy arrays.

These have the advantage that we can avoid having to reproduce all of the logic for when various types turn into other types, which I suspect would be a significant effort to apply to the entire codebase.

Yes, it's a lot of work to add logic for extra types. I don't see any way to avoid that if you want to make a system that works reliably. A good starting point is to assume that existing dask operations only work on arrays containing NumPy arrays, and add support for masked/sparse on a by-exception basis.

Also, another issue, dask.array supports mixing container types.

We could add MixedArray(BaseArray) to my list, or simply use BaseArray for mixed use cases.

[mrocklin]

To me this doesn't seem like a case where types are the right move. Types are hierarchical while this is the container type is only one aspect of many of the array. It feels like a mismatch.

Yes, it's a lot of work to add logic for extra types. I don't see any way to avoid that if you want to make a system that works reliably.

I think that there are many ways to ensure reliability other than types.

[shoyer]

Types are hierarchical while this is the container type is only one aspect of many of the array.

I don't understand. Are you also objecting to the array_type property?

I'm proposing using types (in dask array) to model types (np.ndarray/MaskedArray/SparseArray). It feels pretty natural to me.

I think that there are many ways to ensure reliability other than types.

I think types are one of the very few ways we have to ensure reliable behavior in complex/distributed systems, short of extensive test coverage.

[mrocklin]

I'm not objecting to the array_type property. I'm objecting to subclassing da.Array. I see now though that that isn't what you were suggesting.

MixedArray(BaseArray)

I think that it would be more natural to have tuples of types. Also, I wouldn't build new types like SparseArray, I would just use existing container types like np.ndarray.

I think types are one of the very few ways we have to ensure reliable behavior in complex/distributed systems, short of extensive test coverage.

Lets note that we are now at the point of philosophical discussion where there is no correct or incorrect answer. My opinion is that types are quite valuable, but that they (like tests) don't ensure reliable behavior. They are both valuable, as are a number of other mechanisms.

[mrocklin]

Copying over a comment from duplicate #4070

Sometimes we want to ask questions about the kind of constituent arrays we have within our dask arrays. For example we might want to know if it is sparse or on the GPU.

Currently we are unable to answer these questions, except by doing a small computation.

To resolve this we might consider storing a small empty array rather than the dtype. This is similar to what we do with the _meta attribute to Dask Dataframes.

This would help us to understand more about our arrays, but would also introduce some potential challenges when computing metadata of subsequent arrays.

@shoyer suggested

though it might be enough to store the type of the arrays in each chunk.

I think that for many of the questions we want to ask, storing the type suffices. However I suspect that it is harder to propagate type information along. I suspect that it is easier to create a closed system if you have a full object.

Your use of "type of the arrays in each chunk" reminds me that Dask arrays can be mixed. Maybe this means that _meta would have to be a tuple of objects, one for each type present?

[shoyer]

I'm happy going either way on types vs objects, whichever is easier.

Unfortunately, I suspect making the closed system with actual objects will be a little tricky, due to edge cases with arrays of size 0 or size 1 (which NumPy sometimes treats like scalars). But it's probably worth a try.