Skip to content

Google symptoms to automatically produce full history for new signals #1489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 25, 2022

Conversation

nmdefries
Copy link
Contributor

@nmdefries nmdefries commented Jan 25, 2022

Description

Change logic when handling API metadata so that new signals are automatically produced back to export_start_date.

When introducing new signals, delphi_google_symptoms doesn't produce their full history, just the last ~18 days (based on what data the validator needs). We want new signals to be automatically backfilled, so change handling of the API metadata to be more precise.

Specifically, in the old version, we looked at all google-symptoms signals in metadata to determine how far back to generate data. But this doesn't generalize well. E.g. when we deprecate signals, they will stop getting updated in metadata, but the pipeline will keep trying to update back to the last day they were available for.

This changes metadata processing so that we only look at entries in metadata for currently-generated signals (based on signal names in constants.py). If any signals that we want to generate don't appear in metadata, we assume that they are new signals and produce the full history.

Changelog

  • run.py

@nmdefries nmdefries requested a review from nloliveira January 25, 2022 19:43
Copy link
Contributor

@nloliveira nloliveira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@nmdefries nmdefries merged commit 57ca474 into google_symptoms_omicron Jan 25, 2022
@nmdefries nmdefries deleted the ndefries/gs-new-signals-fixes branch January 25, 2022 20:21
@nmdefries nmdefries restored the ndefries/gs-new-signals-fixes branch January 26, 2022 14:20
@nmdefries nmdefries deleted the ndefries/gs-new-signals-fixes branch February 18, 2022 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants