GHT has misleading declining trend --- in areas with unexpectedly low volume?

[brookslogan]

In MSAs like:

• Beaumont-Port Arthur, TX
• College Station-Bryan, TX
• Sioux City, IA-NE-SD
• Cheyenne, WY

It appears that the GHT signal is encountering single queries or single days with queries above the reporting threshold on scattered days, but then the smoothing/averaging over the following days make it look like a declining trend over the next 7 days. Especially with the default of showing the last two weeks only in the graph, this often appears like a (misleading) rapidly decreasing trend, or strange spikes upward.

(I guess this goes back to real 0s vs. under threshold; not sure what the status on handling these is.) It seems we need to adjust the smoothing to either: skip over missed reporting or perhaps both missed reporting and real 0s --- e.g., if there are only 3 nonmissing days, average over only those 3 with a 3 in the denominator rather than 7 --- or report NAs for the smoothed estimates instead.

[brookslogan]

The second potential issue here is why College Station isn't meeting the reporting threshold most of the time; I would think it's pretty populous but I am not certain.

[krivard]

Yes; this is related to #36 -- current smoothing is designed to handle regions with occasional missingness; it does poorly with areas that are typically missing with only occasional data.

[capnrefsmmat]

College Station is populous during the school year; otherwise it's a pretty small college town.

[brookslogan]

Since we are doing "filtering" --- best estimate for time t using data up to time t (or t + lag) --- I don't think we can avoid spikes upward without sacrificing prediction accuracy or availability. For viz purposes I would think a spike upward, plateau, then spike downward might mislead viewers less (maybe viz team's user studies will give actual data on this), but this might also be at the cost of prediction accuracy. Guess there is not a clear course of action here that meets all purposes simultaneously. Maybe expanding the default viz time scale to include last four weeks could help. But the issue might still remain if a signal is missing for the last 3 weeks in a row.

[huisaddison]

I agree with @brookslogan 's last comment. (I will refer to "filtering := left smoothing" and "smoothing := symmetric smoothing" to avoid overloading the term "smoothing".)

Basically, I devised the smoothing method for GHT, which was required to be a left smoother, to be "as smooth as possible without sacrificing the ability to tell that there was a jump today". This leads to a jump when a point mass appears followed by a taper to zero. I thought that this would be preferable to turning a point mass, say on Monday, to a symmetric mass centered on Wednesday. A left smoother is not capable of turning it into a point mass centered on Monday.

My personal view is that the appropriate thing to do for the map is, to perform left-smoothing for the present and symmetric-smoothing for the past (which is the same as recomputing a smoother over all available data, every day). Then maintain well-documented, separate sources of data that are only left-smoothed for end-users who are using it to construct their own models. (More generally, this fits into the discussion of backfill, etc.).

[brookslogan]

I agree with @huisaddison that symmetric smoothers are more natural to plot. But this may be blocked by the addition of the issue and lag columns; I don't know how far progress is on adding these.

Looking again at the map, though, I am not sure if I have correctly described the nature of the GHT patterns we are seeing. It looks like there can be spikes that are not left-smoothed. (But maybe this is from GHT returning different results for the same day?)

[krivard]

Next:

• Put a prototype centered smoother signal in as a wip signal

[jingjtang]

The new signal is added here. The declining trend would be changed to a "spike upward, plateau, then spike downward" pattern.