Add temporally consistent Query encodings #20
Description
Detailed Description
From the discussion on #19 @JackKelly brought up:
I think the position encoding needs to be 'internally consistent' across each entire example. That is, the position encoding should be consistent across data inputs and the queries so the model can see that, say, the last timestep of the recent history in the input is immediately before the first timestep of the query. And that, in general, the model can see that the timeseries of recent history and the timeseries of queries are two parts of a contiguous timeseries.
To give a concrete example: If the recent history in the input spans 11:00 to 11:55, and the query is for 12:00 to 12:55, then we want the model to see that the first timestep of the query is 5 minutes after the last timestep of the input... if that makes sense?!
One way to do this might be to encode the positions once for all timesteps in the example (i.e. the concatenation of the recent history and the forecast timesteps), and then concatenate the last forecast_timesteps of the position encoding to the queries? Or something like that?! Not sure what's best!
Perhaps this could wait for a future PR though!
This is just a hunch, of course!
And, ultimately, we might want the position encoding to include both the relative position ("the ith element in the array") and the absolute position in time and space ("2pm in South London"). But we can worry about that later :)
Context
This could help the model with better understanding how the future timesteps encoded in the query relate to the input data.
Possible Implementation
Possibly by giving LearnableQuery the number of history timesteps, it could then add the features for the ones in the future? A better implementation might be to pass the Fourier encoding created across all the history and future input data, and then concatenate the future part to the learnable query.