[ENH] enable large data use cases - decouple data input from `pandas`, allow `polars`, `dask`, and/or `spark`

[fkiraly]

A key limitation of current architecture seems to be the reliance on pandas of the input, which limites useability in large data cases.

While torch with appropriate backends should be able to handle large data, pandas as a container choice, in particular the current instantiation which seems to rely on in-memory, will prove to be the bottleneck.

We should therefore consider and implement support for data backends that scale better, such as polars, dask, or spark, and see how easy it is to get the pandas pyarrow integration to work.

Architecturally, I think we should:

• build a more abstract data loader layer
• make pandas one of multiple potential data soft dependencies
• try to prioritize the solution that would provide us with the quickest "impact for time invested"

The key entry point for this extension or refactor is TimeSeriesDataSet, which requires pandas objects to be passed.

[Spinachboul]

@fkiraly
Can I be assigned this issue?
I have a little idea about using rust under the hood for integrating polars as one of the dataset.
Also this might be a bit complex given we are handling multiple backend formats
So this might take some time to maintain consistency across all frameworks

[fkiraly]

@Spinachboul, this is a very complex issue because it relies on at least a redesign of the API.

You can work on this, but first we need an API written down that would allow multiple backends.

Feel free to suggest something, or participate in the discussions in #1736 until we have consolidated this!

We will also use some of the meet-ups for the pytorch-forecasting rework.

[Spinachboul]

@fkiraly Ohk then! I am not quite familiar with the abbreviation DSIPTS, could you please give some reference for it?
Also could you direct me towards what code files to look for amendments in the backend for better understanding as I am unfamiliar with the codebase!!
Thanks!

[fkiraly]

I am not quite familiar with the abbreviation DSIPTS, could you please give some reference for it?

There is a link directly at the top of the mentioned issue #1736

Also could you direct me towards what code files to look for amendments in the backend for better understanding as I am unfamiliar with the codebase!

Primarily BaseModel and TimeSeriesDataSet in pytorch-forecasting.

[Spinachboul]

I am not quite familiar with the abbreviation DSIPTS, could you please give some reference for it?

There is a link directly at the top of the mentioned issue #1736

Also could you direct me towards what code files to look for amendments in the backend for better understanding as I am unfamiliar with the codebase!

Primarily BaseModel and TimeSeriesDataSet in pytorch-forecasting.

oh ok, thanks!!

[benHeid]

One addition from my side. We should at least implement one dataset that is able to read data by itself. So that the user does not need to provide a data object during initialisation. This would also improve or enable usage with large datasets.

[fkiraly]

agree!