Google symptoms to automatically produce full history for new signals #1489
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Change logic when handling API metadata so that new signals are automatically produced back to
export_start_date
.When introducing new signals,
delphi_google_symptoms
doesn't produce their full history, just the last ~18 days (based on what data the validator needs). We want new signals to be automatically backfilled, so change handling of the API metadata to be more precise.Specifically, in the old version, we looked at all
google-symptoms
signals in metadata to determine how far back to generate data. But this doesn't generalize well. E.g. when we deprecate signals, they will stop getting updated in metadata, but the pipeline will keep trying to update back to the last day they were available for.This changes metadata processing so that we only look at entries in metadata for currently-generated signals (based on signal names in
constants.py
). If any signals that we want to generate don't appear in metadata, we assume that they are new signals and produce the full history.Changelog
run.py