New Feature: Add a TidyData Class

[AlexAndorra]

This issue is up for grabs, with the goal of developing a new feature called pm.TidyData. The goal is to deal with automatically translating strings from a dataframe into integer arrays for indexing in models, and then make sure we still get the right labels in the output, for plots and diagnostics.

@aseyboldt implemented a first version and an example, but the implementation is not mature enough, so we decided to not include it in #3551.

This would be a very useful new feature though, which is why we created this issue -- to remind ourselves of it and in case someone wants to give it a try 😉

[NowanIlfideme]

Any reason to not support this explicitly via xarray? The coords and dims mapping between pandas and xarray works pretty well (see below), and xarray already natively has coords and dims (+ ArviZ's InferenceData is a pack of xr.Dataset objects as well). This would help workflows that use multidimensional or hierarchical data, for example.

For pandas use cases conversion should be very easy: df.to_xarray() converts a DataFrame to a Dataset with one or more coordinates (e.g. from a MultiIndex; see docs), and the inverse conversion works as well.

[twiecki]

@NowanIlfideme That sounds like a good approach to me. Want to do a PR?

[AlexAndorra]

Yeah, feel free to open a PR, basing it on @aseyboldt's previous approach or not. ArviZ and xarray evolved a lot since Adrian's first thought about this, so there may be easier approches now. This issue is just a guide to express the need and explicit the goal

[kc611]

@twiecki @AlexAndorra Does this mean that the TidyData logic first added in #3551 should be simply re-implemented via xarray instead of numpy.ndarray. Or were there some features expected from the initial implementation which it did not fulfill ?

[AlexAndorra]

Hi @kc611 ! Mmmh, I think it's more about extending the current implementation to work with xarray datasets (and thus Pandas dataframe), as shown in the example above. That way, you get all the dims, coords and associated indexes defined at the same time and place, and conveniently so.
Of course, my memory of the details of what we did for this in August is a bit fuzzy, and ArviZ/xarray evolved since then, so you may find that another approach is more appropriate if you take that on.
How does that sound?

[kc611]

Alright I'll see if I can come up with something.

Should this class also support implicit conversion for pd.DataFrame objects (or should inputs be only limited to xarray Datasets)?

[AlexAndorra]

Awesome, looking forward to it @kc611 !

Should this class also support implicit conversion for pd.DataFrame objects (or should inputs be only limited to xarray Datasets)?

I think inputs could be both pd.Dataframe and xr.Dataset, but the conversion should not be made by PyMC, it should be made by the user

[twiecki]

I don't think anyone would input xarrays, we only need DataFrames.

[kc611]

@twiecki I have a related draft running(implemented using xarray and ). We could do a .to_xrray() if the input is a pd.Dataframe over there. But if not (and we want the support limited to pd.Dataframes ) then I guess I will have to revert back to the original TidyData class implementation. In any case I think we should try to have support for both since the interconversion between two looks quite seamless.

I think it'll be a better idea if you have a look at it (the draft PR) anyway in it's current state.

[NowanIlfideme]

I don't think anyone would input xarrays, we only need DataFrames.

Disagree completely - I would love to input xarray datasets, in fact for several of my latest models I've had to manually add the values to the model.

Dataframes are good too, but 2d Dataframes are easily converted to xarray, while Multi-Index ones can be trickier (the implementers just needs to make sure that they don't auto-expand the MultiIndex during conversion, ie need to choose the proper converter).

Finally, with arviz integrating with xarray more and more, I think pymc3 doing so would be great as well.