-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Open
Labels
:Analytics/ES|QLAKA ESQLAKA ESQL>refactoringTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)
Description
In the substitutions batch of our LogicalPlanOptimizer, there's 4 rules that take an expression like | STATS foo = avg(x*x) + 2 and turn this into a simple aggregation with enclosing EVALs; in this example, this becomes (essentially)
| EVAL $$x = x*x
| STATS $$foo_sum = sum($$x), $$foo_count = count($$x)
| EVAL $$foo = $$foo_sum/$$foo_count, foo = $$foo + 2
| KEEP foo
This is becoming complicated and more difficult to argue about due to the substitutions happening in 4 rules; let's see if we can do with just 2 rules.
More specifically,
ReplaceStatsNestedExpressionWithEvalturnsSTATS avg(x*x) + 2intoEVAL $$x = x*x | STATS foo = avg($$x) + 2.ReplaceStatsAggExpressionWithEvalthen turns| STATS foo = avg($$x) + 2into| STATS $$foo = avg($$x) | EVAL foo = $$foo + 2SubstituteSurrogatesreplaces| STATS $$foo = avg($$x)by| STATS $$foo_sum = sum($$x), $$foo_count = count($$x) | EVAL $$foo = $$foo_sum/$$foo_count- Then we run
ReplaceStatsNestedExpressionWithEvalagain to account for stuff that happened inTranslateMetricsAggregate
It makes sense that there's 1 rule that creates EVALs after the aggregation (ReplaceStatsNestedExpressionWithEval) and one that pulls nested expressions out of agg functions into an EVAL before the aggregation (ReplaceStatsAggExpressionWithEval).
- However,
SubstituteSurrogatesshould only substitute and letReplaceStatsNestedExpressionWithEvalhandle creating theEVALafter theSTATS. - We should check if we can somehow do without a second run of
ReplaceStatsNestedExpressionWithEvalafterTranslateMetricsAggregate
Metadata
Metadata
Assignees
Labels
:Analytics/ES|QLAKA ESQLAKA ESQL>refactoringTeam:AnalyticsMeta label for analytical engine team (ESQL/Aggs/Geo)Meta label for analytical engine team (ESQL/Aggs/Geo)