simple "sanity-check" datasets for forecasters

[dsweber2]

The idea is to make sure that the forecaster is well-behaved and doesn't have any errors before it hits the main dataset that takes a while to run.
Examples include

• dataset with a constant value
  • dataset with a different constant per-state
• data including NA's in the side data sources
• significant latency on the response
• linearly increasing signal
• differing latency between the response and the covariates (both ways)
• data with states with different latency
• iid poisson noise (should predict a flatline)
  • white noise random walk
  • low count tests in general
• some sort of SIR simulation?
• main value is sidechannel+noise (might not be run on every model)
• confirming that scaled_pop can produce exactly the same result as arx_forecaster
• data source which has blatantly obvious revision errors, like sudden massive spikes returning to a previous lower value
• better version of delay test (hand craft examples of problems, rather than random offsets)
• NA's as the last entry for every state
• Which dates predicted values are available for/making sure that preds are on days where we have enough data
• Right now, looks like predictions are only for dates that are included in the input dataset. Might be useful to check behavior when making true forecasts (again, that output dates are what we expect, etc)

(last two from here)

[dsweber2]

there are some examples in epiprocess

[dsweber2]

make sure there are the right number of predictions based on the aheads and lags

[dsweber2]

2 week training window for 4 week ahead (can cause problems).

generally do sanity checks on amount of training data

[dshemetov]

part of broader question headache: how to handle gaps in data? maybe forward fill if the gap is less than a week in width?

[dsweber2]

The initial batch was added in #32, some more potentials added to the list

[dsweber2]

coming out of Dmitry's experience with flu-hosp: something which has only NA's as the last entry for every state