Refactor smoothers as a utility, create new filters.

[jsharpna]

[krivard]

• Standardize input and output: numpy array with one cell per day
• Start with JHU or another Jingjing & Addison production; that's the most common style amongst the python indicators.

[dshemetov]

I think it may be a good to collect all the possible sources of data missingness and come up with a standard approach:

• missing from source (temporarily or permanently)
• privacy censoring

One approach to missing data can be seen in geo_reindex here. The TL;DR is: we fill a regular grid of daily values with the signal and all the missing values are then filled with 0's.

I wonder if it would be better to mark them with NANs instead. This would make it clear on the smoothing end that the value is missing and not 0. There we can decide to impute the data based on the most recent historic data or geographic factors.

[dshemetov]

Also, we should probably think through smoother boundary effects issues. Currently some smoothers will report NANs for approximately the first few weeks of the available data (the NAN window is based on the averaging window of ~2 weeks). This data will be so far in the past, that it likely won't matter for practical real-time users, but is worth considering for historical use-cases. A natural way to fix this would be to dynamically change the smoothing window on the boundaries.

[krivard]

Will incorporate into JHU first.

Comparing results to the reference implementation can be done in two copies of the repository or between a feature branch and the main branch.

Data censoring occurs within each indicator to handle data that are permitted by the DUA (ie no stderr/sample size) vs minimum sample size. Missingness due to censorship should be NA. Some sources report 0 when it is not necessarily a true 0 (GHT, cases/deaths).

There are a lot of parameters to the smoothers -- at some point we'll want to evaluate different configurations for typical and edge case performance.

[dshemetov]

#177 JHU refactoring is almost done. Want to write a couple better smoother tests first.

[krivard]

Add to other indicators, order TBD

[krivard]

• python template
• ght
• emr-hosp
• quidel covid tests
• doctor-visits [blocked?]
• quidel flu tests [blocked; wait for merge to main]
• jhu [blocked; wait for geo revision]
• usafacts [blocked; wait for geo revision]
• combo cases/deaths [blocked; wait for diff revision]
• combo nmf_day_doc_fbc_fbs_ght [blocked; not in this codebase]
• fb-survey [blocked; wait for merge to main]