-
Notifications
You must be signed in to change notification settings - Fork 8
Closed
Labels
P2low prioritylow priorityREPLImproved print, errors, etc.Improved print, errors, etc.op-semanticsOperational semantics; many potentially breaking changes hereOperational semantics; many potentially breaking changes hereperformance
Description
Currently, we do not check for distinct key values in epi_archive.
- User experience-wise: this is convenient but also gives them the opportunity to form unexpected or invalid archives.
- Semantics: we assume there aren't duplicate key values.
$as_ofmight give the right result if the duplicate keys have duplicate values and we don't want duplicate rows as output, but gives the wrong thing if instead user is trying to take advantage of a particular update-reporting structure which might enable some performance improvements, which we could take further advantage of by being more flexible with the key, described next. - Performance: we require a version-search for every key value across the single huge archive DT.
- If all (geo_value, time_value, otherkey1, ..., otherkeyn) are re-reported in every
version--- we are working off of full snapshots in DT --- then we only need to look up the version once, and can key byversionalone. But we can't use theuniquelookup foras_ofhere; maybe a rolling join would work & generalize to the next case. - If all (geo_value, otherkey1, ..., otherkeyn) are re-reported in every
versionfor time_valuesversion - 1:windowlength, then we could key just by (time_value, version). - If we have patch-based reporting and no special guarantees, then we need to have (geo_value, otherkey1, ..., otherkeyn, version) as the key, and there should be no duplicates.
- (Stratifying the DT into multiple DTs, say, one per geo, might increase the number of lookups required but make them faster.)
- If all (geo_value, time_value, otherkey1, ..., otherkeyn) are re-reported in every
Might interact with #87.
Metadata
Metadata
Assignees
Labels
P2low prioritylow priorityREPLImproved print, errors, etc.Improved print, errors, etc.op-semanticsOperational semantics; many potentially breaking changes hereOperational semantics; many potentially breaking changes hereperformance