You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Forecast Evaluation Research Collaborative was founded by the [Reich Lab](https://reichlab.io/) at University of Massachusetts Amherst and the Carnegie Mellon University [Delphi Group](https://delphi.cmu.edu). Both groups are funded by the CDC as Centers of Excellence for Influenza and COVID-19 Forecasting. We have partnered together on this project to focus on providing a robust set of tools and methods for evaluating the performance of epidemic forecasts.
4
+
5
+
The collaborative’s mission is to help epidemiological researchers gain insights into the performance of their forecasts, and ultimately lead to more accurate forecasting of epidemics.
6
+
7
+
Both groups have led initiatives related to COVID-19 data and forecast curation. The Reich Lab has created the [COVID-19 Forecast Hub](https://covid19forecasthub.org/), a collaborative effort with over 80 groups submitting forecasts to be part of the official [CDC COVID-19 ensemble forecast](https://www.cdc.gov/coronavirus/2019-ncov/covid-data/mathematical-modeling.html). The Delphi Group has created COVIDcast, a platform for [epidemiological surveillance data](https://delphi.cmu.edu/covidcast/), and runs the [Delphi Pandemic Survey via Facebook](https://delphi.cmu.edu/covidcast/surveys/), which is a [valuable signal](https://delphi.cmu.edu/blog/2020/09/21/can-symptoms-surveys-improve-covid-19-forecasts/) for Delphi’s participation in the ensemble forecast.
8
+
9
+
The Forecaster Evaluation Dashboard is a collaborative project, which has been made possible by the 13 pro bono Google.org Fellows who have spent 6 months working full-time with the Delphi Group. Google.org is [committed](https://www.google.org/covid-19/) to the recovery of lives and communities that have been impacted by COVID-19 and investing in developing the science to mitigate the damage of future pandemics.
10
+
11
+
#### **Collaborators**
12
+
13
+
From the Forecast Hub: Estee Cramer, NIcholas Reich, [the COVID-19 Forecast Hub Team](https://covid19forecasthub.org/doc/team/)
14
+
From the Delphi Research Group: Jed Grabman, Kate Harwood, Chris Scott, Jacob Bien, Daniel McDonald, Logan Brooks
15
+
16
+
### About the Data
17
+
18
+
#### **Sources**
19
+
20
+
**Observed values** are from the [COVID-19 Data Repository](https://github.com/CSSEGISandData/COVID-19) by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University.
21
+
22
+
**Forecaster predictions** are drawn from the [COVID-19 Forecast Hub GitHub repository](https://github.com/reichlab/covid19-forecast-hub/)
23
+
24
+
Data for the dashboard is pulled once a week from these sources, on Tuesdays.
25
+
26
+
#### **Terms**
27
+
28
+
***Forecaster**
29
+
30
+
A model producing quantile predictions
31
+
32
+
***Forecast**
33
+
34
+
A set of data that, for all locales in a geo type, includes predictions for a target variable for each of a certain number of quantiles for each of a certain number of horizons
35
+
36
+
***Target Variable**
37
+
38
+
What the forecast is predicting, ie: “weekly incident cases”
39
+
40
+
***Horizon**
41
+
42
+
The duration of time between when the prediction was made and the predicted event, typically in units of epidemiological weeks.
43
+
44
+
***Epidemiological Week (Epi-week)**
45
+
46
+
Week that starts on a Sunday. If it is Sunday or Monday, the next epi-week is the week that starts on that Sunday (going back a day if it is Monday). If it is Tuesday-Saturday, it is the week that starts on the subsequent Sunday, following [CDC convention](https://wwwn.cdc.gov/nndss/document/MMWR_week_overview.pdf).
47
+
48
+
***Point Forecast**
49
+
50
+
The value that each forecaster picks as their “most likely” prediction. For many forecasters this is the 50th quantile of the predictive distribution, for others it might be the mean of the distribution.
51
+
52
+
***Geo Type**
53
+
54
+
States or U.S. as a nation
55
+
56
+
#### **Dashboard Inclusion Criteria**
57
+
58
+
* Includes only weekly deaths incidence and weekly case incidence target variables
59
+
* Includes only horizon < 5 weeks ahead
60
+
* Includes only geo values that are 2 characters (states / territories / nation)
61
+
* Includes only non-NA target dates (if the date is not in yyyy/mm/dd, the prediction will not be included)
62
+
* Includes only predictions with at least 3 quantile values
63
+
* Includes only one file per forecaster per week (according to forecast date). That file must be from a Sunday or Monday. If both are present, we keep the Monday data.
64
+
* If a forecaster updates a file after that Monday, we do not include the new predictions
65
+
66
+
#### **Notes on the Data**
67
+
68
+
* When totaling over all locations, these locations include states and territories and do not include nationwide forecasts. We only include states and territories common to the selected forecasters (over all time) that have data for at least one location.
69
+
* We do include revisions of observed values, meaning the scores for forecasts made in the past can change. Scores change as our understanding of the truth changes.
The **absolute error** of a forecast is calculated from the Point Forecast. Usually this is the 50% quantile prediction, but forecasters can specify their own Point Forecast value. When none is provided explicity, we use the 50% quantile prediction.
"All forecasts are evaluated against the latest version of observed data. Scores of pasts forecasts may change as observed data is revised."
98
80
99
81
# About page content
100
-
aboutPageText= HTML("
101
-
<div style='width: 80%'>
102
-
<b><h3><u>Who We Are</u></h3></b><br>
103
-
The Forecast Evaluation Research Collaborative was founded by the <a href='https://reichlab.io/'>Reich Lab</a>
104
-
at University of Massachusetts Amherst and the Carnegie Mellon University <a href = 'https://delphi.cmu.edu'> Delphi Group</a>.
105
-
Both groups are funded by the CDC as Centers of Excellence for Influenza and COVID-19 Forecasting.
106
-
We have partnered together on this project to focus on providing a robust set of tools and methods for evaluating the performance of epidemic forecasts.
107
-
<br><br>
108
-
The collaborative’s mission is to help epidemiological researchers gain insights into the performance of their forecasts,
109
-
and ultimately lead to more accurate forecasting of epidemics.
110
-
<br><br>
111
-
Both groups have led initiatives related to COVID-19 data and forecast curation.
112
-
The Reich Lab has created the <a href='https://covid19forecasthub.org/'>COVID-19 Forecast Hub</a>,
113
-
a collaborative effort with over 80 groups submitting forecasts to be part of the official
The Delphi Group has created COVIDcast, a platform for <a href='https://delphi.cmu.edu/covidcast/'>epidemiological surveillance data</a>,
116
-
and runs the <a href='https://delphi.cmu.edu/covidcast/surveys/'>Delphi Pandemic Survey via Facebook</a>,
117
-
which is a <a href='https://delphi.cmu.edu/blog/2020/09/21/can-symptoms-surveys-improve-covid-19-forecasts/'>valuable signal</a>
118
-
for Delphi’s participation in the ensemble forecast.
119
-
<br><br>
120
-
The Forecaster Evaluation Dashboard is a collaborative project, which has been made possible by the 13 pro bono Google.org Fellows
121
-
who have spent 6 months working full-time with the Delphi Group.
122
-
Google.org is <a href='https://www.google.org/covid-19/'>committed</a> to the recovery of lives
123
-
and communities that have been impacted by COVID-19 and investing in developing the science to mitigate the damage of future pandemics.
124
-
<br><br>
125
-
<br><h4><b>Collaborators</b></h4>
126
-
<br>
127
-
From the Forecast Hub: Estee Cramer, NIcholas Reich, <a href='https://covid19forecasthub.org/doc/team/'>the COVID-19 Forecast Hub Team</a>
128
-
<br>
129
-
From the Delphi Research Group: Jed Grabman, Kate Harwood, Chris Scott, Jacob Bien, Daniel McDonald, Logan Brooks
130
-
<br><br><br><b><h3><u>About the Data</u></h3></b>
131
-
<br><h4><b>Sources</b></h4>
132
-
<b>Observed values</b> are from the
133
-
<a href='https://github.com/CSSEGISandData/COVID-19'>COVID-19 Data Repository</a>
134
-
by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University.
135
-
<br><br><b>Forecaster predictions</b> are drawn from the <a href='https://github.com/reichlab/covid19-forecast-hub/'>COVID-19 Forecast Hub GitHub repository</a>
136
-
<br><br>Data for the dashboard is pulled once a week from these sources, on Tuesdays.
137
-
<br><br>
138
-
<h4><b>Terms</b></h4>
139
-
<ul><li><b>Forecaster</b>
140
-
<div style = 'margin-left:40px;'>A model producing quantile predictions</div></li>
141
-
<li><b>Forecast</b>
142
-
<div style = 'margin-left:40px;'>A set of data that, for all locales in a geo type,
143
-
includes predictions for a target variable for each of a certain number of quantiles
144
-
for each of a certain number of horizons </div></li>
145
-
<li><b>Target Variable</b>
146
-
<div style = 'margin-left:40px;'>What the forecast is predicting, ie: “weekly incident cases”</div></li>
147
-
<li><b>Horizon</b>
148
-
<div style = 'margin-left:40px;'>
149
-
The duration of time between when the prediction was made and the predicted event, typically in units of epidemiological weeks.
150
-
</div></li>
151
-
<li><b>Epidemiological Week (Epi-week)</b>
152
-
<div style = 'margin-left:40px;'>Week that starts on a Sunday. If it is Sunday or Monday,
153
-
the next epi-week is the week that starts on that Sunday (going back a day if it is Monday).
154
-
If it is Tuesday-Saturday, it is the week that starts on the subsequent Sunday, following
<div style = 'margin-left:40px;'>The value that each forecaster picks as their “most likely” prediction.
159
-
For many forecasters this is the 50th quantile of the predictive distribution, for others it might be the mean of the distribution.
160
-
</div></li>
161
-
<li><b>Geo Type</b>
162
-
<div style = 'margin-left:40px;'>States or U.S. as a nation</div></li></ul>
163
-
<br><h4><b>Dashboard Inclusion Criteria</b></h4>
164
-
<ul>
165
-
<li> Includes only weekly deaths incidence and weekly case incidence target variables</li>
166
-
<li> Includes only horizon < 5 weeks ahead</li>
167
-
<li> Includes only geo values that are 2 characters (states / territories / nation)</li>
168
-
<li> Includes only non-NA target dates (if the date is not in yyyy/mm/dd, the prediction will not be included)</li>
169
-
<li> Includes only predictions with at least 3 quantile values</li>
170
-
<li> Includes only one file per forecaster per week (according to forecast date). That file must be from a Sunday or Monday. If both are present, we keep the Monday data.</li>
171
-
<li> If a forecaster updates a file after that Monday, we do not include the new predictions</li>
172
-
</ul>
173
-
<br><h4><b>Notes on the Data</b></h4>
174
-
<ul>
175
-
<li>When totaling over all locations, these locations include states and territories and do not include nationwide forecasts.
176
-
We only include states and territories common to the selected forecasters (over all time) that have data for at least one location.</li>
177
-
<li>We do include revisions of observed values, meaning the scores for forecasts made in the past can change.
178
-
Scores change as our understanding of the truth changes.</li>
179
-
</ul>
180
-
<br><br>
181
-
<b><h3><u>Explanation of Scoring Methods</u></h3></b>
The **coverage plot** shows how well a forecaster's confidence intervals performed on a given week, across all locations. The horizontal black line is the selected confidence interval, and the y-values are the percentage of time that the observed values of the target variable value fell into that confidence interval. A perfect forecaster on this measure would follow the black line.
2
+
3
+
For example, a forecaster wants the observed values to be within the 50% confidence interval in 50% of locations for the given week. If the y-value is above the horizontal line, it means that the observed values fell within the forecaster's 50% CI more than 50% of the time, aka the forecaster's 50% CI was under-confident that week, or too wide. Conversely, if the y-value is below the line, it means that the forecaster's 50% CI was over-confident that week, or too narrow.
The **weighted interval score** (WIS) is a proper score that combines a set of interval scores. See [this article](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008618) about the WIS method for a more in depth explanation. The WIS factors in both the sharpness of prediction intervals and their calibration (or coverage) of the actual observations.
0 commit comments