-
Notifications
You must be signed in to change notification settings - Fork 8
Closed
Labels
P2low prioritylow priorityREPLImproved print, errors, etc.Improved print, errors, etc.op-semanticsOperational semantics; many potentially breaking changes hereOperational semantics; many potentially breaking changes hereperformance
Description
Currently, we do not check for distinct key values in epi_archive
.
- User experience-wise: this is convenient but also gives them the opportunity to form unexpected or invalid archives.
- Semantics: we assume there aren't duplicate key values.
$as_of
might give the right result if the duplicate keys have duplicate values and we don't want duplicate rows as output, but gives the wrong thing if instead user is trying to take advantage of a particular update-reporting structure which might enable some performance improvements, which we could take further advantage of by being more flexible with the key, described next. - Performance: we require a version-search for every key value across the single huge archive DT.
- If all (geo_value, time_value, otherkey1, ..., otherkeyn) are re-reported in every
version
--- we are working off of full snapshots in DT --- then we only need to look up the version once, and can key byversion
alone. But we can't use theunique
lookup foras_of
here; maybe a rolling join would work & generalize to the next case. - If all (geo_value, otherkey1, ..., otherkeyn) are re-reported in every
version
for time_valuesversion - 1:windowlength
, then we could key just by (time_value, version). - If we have patch-based reporting and no special guarantees, then we need to have (geo_value, otherkey1, ..., otherkeyn, version) as the key, and there should be no duplicates.
- (Stratifying the DT into multiple DTs, say, one per geo, might increase the number of lookups required but make them faster.)
- If all (geo_value, time_value, otherkey1, ..., otherkeyn) are re-reported in every
Might interact with #87.
Metadata
Metadata
Assignees
Labels
P2low prioritylow priorityREPLImproved print, errors, etc.Improved print, errors, etc.op-semanticsOperational semantics; many potentially breaking changes hereOperational semantics; many potentially breaking changes hereperformance