Lazy apply

[mrocklin]

Add function to lazily apply a function onto a materialized dask array. Came up in conversation with @shoyer

I'm not sure how I feel about this yet. I'm afraid that people will assume it does something intelligent and misuse it. I kind of what to hide this some place. Thoughts?

[shoyer]

I didn't realize that you could compose functions in dask graphs by nesting tuples, but I guess that makes sense.

[shoyer]

Would another option be to do something like the following here?

consolidated = x.reblock(blockshape=x.shape)
dsk = {(name,) + (0,) * len(shape): (func, consolidated.name)}
return Array(merge(dsk, consolidated.dask), name, blockdims=blockdims, dtype=dtype)

That seems slightly preferable to using the private _keys method.

[shoyer]

I should show you how far I got on lazy_apply for xray.Dataset objects. It's messier, because I need to store a function that creates an xray.Dataset in the dask.

[mrocklin]

@shoyer I've removed lazy_apply from the main namespace, so you'll need to dive more deeply into da.core.lazy_apply. Are you ok with this? Thoughts generally on this kind of thing? This is ready to merge from my perspective.

[shoyer]

@mrocklin That's fine by me. This is more valuable to me as an example, anyways, because it doesn't quite do what I need (which will require multiple dask arrays as input). I don't think I'll be able to create a lazy_apply which works on xray.Dataset objects without actually manipulating the dask in xray (which is fine).

[mrocklin]

OK, if this isn't directly usable then my preference is not to merge it. If I can do something usable then let me know. We could do a lazy_apply that took many input arrays if that's better.

[shoyer]

A version that took many input arrays and returned many output arrays would be directly usable.

I wrote a version of this function that acts directly on a lazy xray dataset as input. That works in terms of not blowing up my memory but the "dask within a dask" approach is obviously not ideal -- it kind of cripples the scheduler.

[mrocklin]

Stale. Closing. Please reopen on renewed interest.