ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut #135011

astefan · 2025-09-18T15:06:48Z

This PR covers a small range of edge cases where the logical optimization rules "compete" in optimizing aggregations to a simplified tree that is either a LocalRelation by itself or that have the source leaf node as LocalRelation. When the right-hand side becomes a non-concrete source, it is one of two situations now:

the EsqlSession ran the left-hand side query and the results are modeled as LocalRelation which replace the StubRelation of the right-hand side
the logical optimization flow optimized away the right-hand side (essentially transformed it into a local query, no ES data involved)

The code in the PR covers the second scenario above. There are few rules which touch the idea of inline stats and which optimize away the right-hand side, but there are also others which are generic and have no different behavior for inline stats.

PruneEmptyPlans is one rule where I debated a lot if it should also deal with StubRelation as an "empty" plan or remain as is. I decided that the particularities of inlinejoin are not the concern of this rule, but those of the EsqlSession where the "coordination" of the inlinejoin is mostly happening.
PruneColumns is another rule where the pruning of the right-hand side is handled
lastly, ReplaceStatsFilteredAggWithEval deals with pruning inline stats and stats that have filters that can also be pruned

LocalRelation

elasticsearchmachine · 2025-09-18T15:07:14Z

Hi @astefan, I've created a changelog YAML for you.

…handle_right_LocalRelation

elasticsearchmachine · 2025-09-19T08:26:53Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

astefan · 2025-09-22T14:21:55Z

...in/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PruneColumns.java

+    private static LogicalPlan emptyLocalRelation(UnaryPlan plan) {
        // create an empty local relation with no attributes
-        return new LocalRelation(plan.source(), plan.output(), EmptyLocalSupplier.EMPTY);
+        return skipPlan(plan);


Unrelated to the meat of the PR

…handle_right_LocalRelation

bpintea

LG, not sure about the source of the failure, but it seems it might have a different origin? Happy to re-review otherwise.

bpintea · 2025-09-22T19:45:07Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/InlineJoin.java

+                } else if (ij.right() instanceof LocalRelation && skipped == null
+                    || ij.right() instanceof LocalRelation == false && ij.right().anyMatch(p -> p instanceof LocalRelation)) {


Could this be:

Suggested change

} else if (ij.right() instanceof LocalRelation && skipped == null

|| ij.right() instanceof LocalRelation == false && ij.right().anyMatch(p -> p instanceof LocalRelation)) {

} else if (ij.right().children().stream().anyMatch(c -> c.anyMatch(LocalRelation.class::isInstance))) {

Then I think you wouldn't need to pass the extra skipped param.
Which, otherwise, could be a bool, maybe, since it's only used in a test against null.

bpintea · 2025-09-22T19:47:27Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/InlineJoin.java

+                        var p = ij.right();
+                        p.setOptimized();
+                        subPlan.set(new LogicalPlanTuple(p, ij.right()));


Don't know if it'd be worth creating a light static method to do this here and the similar thing above.

astefan · 2025-09-23T07:29:56Z

@bpintea thank you for having a look. I have added a more detailed description to the PR, trying to clarify the situations (conceptually) this PR is trying to address and why this particular fix has been chosen. Hope this helps.

…handle_right_LocalRelation

bpintea · 2025-09-25T15:47:38Z

...c/main/java/org/elasticsearch/xpack/esql/optimizer/rules/PruneEmptyInlineStatsRightSide.java

+import org.elasticsearch.xpack.esql.plan.logical.join.InlineJoin;
+import org.elasticsearch.xpack.esql.plan.logical.local.LocalRelation;
+
+public final class PruneEmptyInlineStatsRightSide extends OptimizerRules.OptimizerRule<InlineJoin> {


We have a PruneLeftJoinOnNullMatchingField. Should we call this some similar way? (maybe PruneInlineJoinOnEmptyRightSide -- I guess we could use InlineJoin instead of InlineStats, since the rule should apply correctly even if we found other uses of InlineJoin).

Yep, makes sense. I've renamed it.

bpintea · 2025-09-25T16:20:57Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/join/InlineJoin.java

+                } else if (ij.right() instanceof LocalRelation relation
+                    && (subPlansResults.isEmpty() || subPlansResults.contains(relation) == false)
+                    || ij.right() instanceof LocalRelation == false && ij.right().anyMatch(p -> p instanceof LocalRelation)) {


Might be worth extracting this check into a method, it's a bit hard to read.
I tried to simplify it a bit, but we run into block tracking issues, so I guess this could probably be simplified, with some extra accounting.
But that can be a followup.

…handle_right_LocalRelation

alex-spies · 2025-09-26T10:19:04Z

Late to the party, but just wanted to say this looks right to me. Thanks for keeping me in the loop, the inline stats machinery and how it evolves is super interesting!

…-dls * upstream/main: (55 commits) Mute org.elasticsearch.upgrades.MatchOnlyTextRollingUpgradeIT testIndexing {upgradedNodes=1} elastic#135525 Address es819 tsdb doc values format performance bug (elastic#135505) Remove obsolete --add-opens from JDK API extractor tool (elastic#135445) [CI] Fix MergeWithFailureIT (elastic#135447) Increase wait time in AdaptiveAllocationsScalerServiceTests (elastic#135510) ES|QL: Handle multi values in FUSE (elastic#135448) Mute org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT testIndexing {upgradedNodes=1} elastic#135512 Add trust configuration for cross cluster api keys (elastic#134893) ESQL: Fix flakiness in SessionUtilsTests (elastic#135375) Mute org.elasticsearch.upgrades.LogsdbIndexingRollingUpgradeIT testIndexing {upgradedNodes=1} elastic#135511 Mute org.elasticsearch.upgrades.MatchOnlyTextRollingUpgradeIT testIndexing {upgradedNodes=2} elastic#135325 Require all functions to provide examples (elastic#135094) Mute org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT testIndexing {upgradedNodes=2} elastic#135344 Mute org.elasticsearch.upgrades.TextRollingUpgradeIT testIndexing {upgradedNodes=1} elastic#135236 Mute org.elasticsearch.upgrades.TextRollingUpgradeIT testIndexing {upgradedNodes=2} elastic#135238 Mute org.elasticsearch.upgrades.LogsdbIndexingRollingUpgradeIT testIndexing {upgradedNodes=2} elastic#135327 Mute org.elasticsearch.upgrades.MatchOnlyTextRollingUpgradeIT testIndexing {upgradedNodes=3} elastic#135324 Mute org.elasticsearch.upgrades.StandardToLogsDbIndexModeRollingUpgradeIT testLogsIndexing {upgradedNodes=3} elastic#135315 ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut (elastic#135011) Mute org.elasticsearch.upgrades.TextRollingUpgradeIT testIndexing {upgradedNodes=3} elastic#135237 ...

Handle right hand side of Inline Stats coming optimized with

0f4cb80

LocalRelation

astefan added >bug :Analytics/ES|QL AKA ESQL v9.2.0 labels Sep 18, 2025

Update docs/changelog/135011.yaml

0fc78fb

astefan added 5 commits September 18, 2025 23:56

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

c7204f0

…handle_right_LocalRelation

update

e05190c

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

836235f

…handle_right_LocalRelation

Add capability

109e856

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

912b968

…handle_right_LocalRelation

astefan requested review from alex-spies and bpintea September 19, 2025 08:26

astefan marked this pull request as ready for review September 19, 2025 08:26

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 19, 2025

astefan commented Sep 22, 2025

View reviewed changes

astefan added 3 commits September 22, 2025 20:23

More tests and more edge cases

6b4952c

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

135acf9

…handle_right_LocalRelation

tweaks

4c41f21

bpintea approved these changes Sep 22, 2025

View reviewed changes

astefan added 4 commits September 24, 2025 16:49

Prune "empty" (LocalRelation only) right hand side of Inlinejoin

ba52bd0

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

da6f6cb

…handle_right_LocalRelation

add capability

b57f4dd

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

3e64b2f

…handle_right_LocalRelation

bpintea self-requested a review September 25, 2025 14:30

bpintea approved these changes Sep 25, 2025

View reviewed changes

astefan added 2 commits September 25, 2025 19:55

Address review

8f90299

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

325ad7b

…handle_right_LocalRelation

astefan added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Sep 25, 2025

astefan and others added 3 commits September 26, 2025 08:37

Add import

e877281

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

7518784

…handle_right_LocalRelation

Merge branch 'main' into handle_right_LocalRelation

0c63ba9

elasticsearchmachine merged commit 9dfc8d4 into elastic:main Sep 26, 2025
34 checks passed

astefan deleted the handle_right_LocalRelation branch September 26, 2025 06:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut #135011

ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut #135011

Uh oh!

astefan commented Sep 18, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Sep 18, 2025

Uh oh!

elasticsearchmachine commented Sep 19, 2025

Uh oh!

astefan Sep 22, 2025

Uh oh!

bpintea left a comment

Uh oh!

bpintea Sep 22, 2025 •

edited

Loading

Uh oh!

bpintea Sep 22, 2025

Uh oh!

astefan commented Sep 23, 2025

Uh oh!

bpintea Sep 25, 2025

Uh oh!

astefan Sep 25, 2025

Uh oh!

bpintea Sep 25, 2025

Uh oh!

Uh oh!

alex-spies commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		} else if (ij.right() instanceof LocalRelation && skipped == null
		\|\| ij.right() instanceof LocalRelation == false && ij.right().anyMatch(p -> p instanceof LocalRelation)) {

	} else if (ij.right() instanceof LocalRelation && skipped == null
	\|\| ij.right() instanceof LocalRelation == false && ij.right().anyMatch(p -> p instanceof LocalRelation)) {
	} else if (ij.right().children().stream().anyMatch(c -> c.anyMatch(LocalRelation.class::isInstance))) {

ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut #135011

ESQL: Handle right hand side of Inline Stats coming optimized with LocalRelation shortcut #135011

Uh oh!

Conversation

astefan commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 18, 2025

Uh oh!

elasticsearchmachine commented Sep 19, 2025

Uh oh!

astefan Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

bpintea Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

astefan commented Sep 23, 2025

Uh oh!

bpintea Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

astefan Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

bpintea Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

astefan commented Sep 18, 2025 •

edited

Loading

bpintea Sep 22, 2025 •

edited

Loading