[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

cxzl25 · 2024-08-23T09:25:34Z

Original author: @amuraru

What changes were proposed in this pull request?

This PR aims to support add memory based thresholds for shuffle spill.

Introduce configuration

spark.shuffle.spill.maxRecordsSizeForSpillThreshold
spark.sql.windowExec.buffer.spill.size.threshold
spark.sql.sessionWindow.buffer.spill.size.threshold
spark.sql.sortMergeJoinExec.buffer.spill.size.threshold
spark.sql.cartesianProductExec.buffer.spill.size.threshold

Why are the changes needed?

We can only determine the number of spills by configuring spark.shuffle.spill.numElementsForceSpillThreshold. In some scenarios, the size of a row may be very large in the memory.

Does this PR introduce any user-facing change?

No

How was this patch tested?

GA

Verified in the production environment, the task time is shortened, the number of spill disks is reduced, there is a better chance to compress the shuffle data, and the size of the spill to disk is also significantly reduced.

Current

24/08/19 07:02:54,947 [Executor task launch worker for task 0.0 in stage 53.0 (TID 1393)] INFO ShuffleExternalSorter: Thread 126 spilling sort data of 62.0 MiB to disk (11490  times so far)
24/08/19 07:02:55,029 [Executor task launch worker for task 0.0 in stage 53.0 (TID 1393)] INFO ShuffleExternalSorter: Thread 126 spilling sort data of 62.0 MiB to disk (11491  times so far)
24/08/19 07:02:55,093 [Executor task launch worker for task 0.0 in stage 53.0 (TID 1393)] INFO ShuffleExternalSorter: Thread 126 spilling sort data of 62.0 MiB to disk (11492  times so far)
24/08/19 07:08:59,894 [Executor task launch worker for task 0.0 in stage 53.0 (TID 1393)] INFO Executor: Finished task 0.0 in stage 53.0 (TID 1393). 7409 bytes result sent to driver

PR

Was this patch authored or co-authored using generative AI tooling?

No

HyukjinKwon · 2024-08-26T02:05:08Z

Let's probably file a new JIRA

dongjoon-hyun · 2024-09-11T15:27:22Z

Gentle ping, @cxzl25 and @mridulm .

Although we have enough time until Feature Freeze, I'm wondering if we can deliver this via Apache Spark 4.0.0-preview2 RC1 (next Monday). WDYT?

mridulm · 2024-09-12T05:19:20Z

I am a bit swamped unfortunately, and I dont think I will be able to ensure this gets merged before next monday @dongjoon-hyun - sorry about that :-(

@cxzl25, will try to get around to reviewing this soon - apologies for the delay

mridulm · 2024-09-12T05:19:36Z

+CC @Ngone51 as well.

dongjoon-hyun · 2024-09-12T15:06:54Z

Thank you for letting me know, @mridulm ~ No problem at all.

pan3793 · 2025-04-18T10:02:08Z

Kindly ping @mridulm, do you have a chance to take another look? I also found this PR is helpful for stability for jobs that spill huge data.

mridulm

Just a few comments, mostly looks good to me.
Thanks for working on this @cxzl25, and apologies for the delay in getting to this !

+CC @HyukjinKwon, @cloud-fan as well for review.

core/src/main/scala/org/apache/spark/internal/config/package.scala

mridulm · 2025-04-20T01:36:52Z

core/src/main/scala/org/apache/spark/util/collection/Spillable.scala

By moving _elementsRead > numElementsForceSpillThreshold here, we would actually reduce some unnecessary allocations .... nice !

core/src/main/scala/org/apache/spark/util/collection/Spillable.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

mridulm · 2025-04-20T01:55:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

The config name is a bit confusing.
spark.sql.windowExec.buffer.spill.threshold vs spark.sql.windowExec.buffer.spill.size.threshold.

Same for the others introduced.

I will let @HyukjinKwon or @cloud-fan comment better though.

I am not super used to this area. I would rarther follow the suggestions from you / others.

Thanks @HyukjinKwon !
+CC @dongjoon-hyun as well.

mridulm · 2025-05-03T03:18:23Z

I am planning to merge this next week if there are no concerns @cloud-fan , @dongjoon-hyun.
It has been open for quite a while, and is a very helpful fix to mitigate memory issues.

I am not super keen on the naming of some of the sql configs, would your thoughts on that (as well as rest of the PR).

Also, +CC @attilapiros for feedback as well.

core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java

core/src/main/scala/org/apache/spark/util/collection/Spillable.scala

rahil-c · 2025-06-24T21:27:55Z

@mridulm @cxzl25 @attilapiros @HyukjinKwon @pan3793

Hi all was just curious if there was any issues regarding this pr or if it will be merged in OSS Spark sometime soon? Thanks again for making this change!

mridulm · 2025-06-25T17:09:13Z

I did not merge it given @attilapiros was actively reviewing it.
Are there any other concerns/comments on this Attila ?

attilapiros · 2025-06-25T17:34:44Z

checking

attilapiros

LGTM after the code duplicate is resolved.

sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactory.scala

When running large shuffles (700TB input data, 200k map tasks, 50k reducers on a 300 nodes cluster) the job is regularly OOMing in map and reduce phase. IIUC ShuffleExternalSorter (map side) and ExternalAppendOnlyMap and ExternalSorter (reduce side) are trying to max out the available execution memory. This in turn doesn't play nice with the Garbage Collector and executors are failing with OutOfMemoryError when the memory allocation from these in-memory structure is maxing out the available heap size (in our case we are running with 9 cores/executor, 32G per executor) To mitigate this, one can set spark.shuffle.spill.numElementsForceSpillThreshold to force the spill on disk. While this config works, it is not flexible enough as it's expressed in number of elements, and in our case we run multiple shuffles in a single job and element size is different from one stage to another. This patch extends the spill threshold behaviour and adds two new parameters to control the spill based on memory usage: - spark.shuffle.spill.map.maxRecordsSizeForSpillThreshold - spark.shuffle.spill.reduce.maxRecordsSizeForSpillThreshold

…e and GlutenJoinSuite tests see apache/spark#47856

…ite and GlutenJoinSuite tests see apache/spark#47856

github-actions bot added SQL STRUCTURED STREAMING CORE PYTHON labels Aug 23, 2024

cxzl25 changed the title ~~[SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill~~ [SPARK-49386][SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill Aug 26, 2024

cxzl25 mentioned this pull request Sep 11, 2024

[SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill #24618

Closed

cxzl25 force-pushed the SPARK-27734 branch from 726f800 to b23a4c8 Compare November 6, 2024 07:19

cxzl25 force-pushed the SPARK-27734 branch from b23a4c8 to 781eb4b Compare December 19, 2024 03:17

cxzl25 force-pushed the SPARK-27734 branch from 781eb4b to 22e551c Compare January 17, 2025 03:20

cxzl25 force-pushed the SPARK-27734 branch from 22e551c to 09a9a9c Compare February 14, 2025 07:23

mridulm reviewed Apr 20, 2025

View reviewed changes

cxzl25 force-pushed the SPARK-27734 branch from 09a9a9c to 9962e07 Compare April 21, 2025 04:01

attilapiros reviewed May 3, 2025

View reviewed changes

core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java Outdated Show resolved Hide resolved

attilapiros reviewed May 3, 2025

View reviewed changes

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java Outdated Show resolved Hide resolved

attilapiros reviewed May 4, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/util/collection/Spillable.scala Outdated Show resolved Hide resolved

cxzl25 force-pushed the SPARK-27734 branch from 95fc754 to 043dedb Compare May 26, 2025 04:06

attilapiros approved these changes Jun 25, 2025

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala Outdated Show resolved Hide resolved

sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactory.scala Outdated Show resolved Hide resolved

amuraru and others added 3 commits June 26, 2025 15:40

spill by size

044a232

config

27c6478

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

7fea38b

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

ee1c32a

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

9f42869

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

f83003d

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

1239e45

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

fae9090

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

c9e00a9

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

3b0833e

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

51c010d

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

c791eed

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[TMP] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuit…

9f5a223

…e and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

58500df

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 23, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

90be2da

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 24, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

04dd8ae

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 25, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

07aea7f

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 25, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

be75e12

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 26, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

92ddcec

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 26, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

8c971e1

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 29, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

79d9f43

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 30, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

5ddbfa9

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

0e051f2

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

3805c52

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

9bfe79e

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

bff1810

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

a0183c9

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

7d72059

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

04c61d0

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

c8809b8

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

70a72a9

…ite and GlutenJoinSuite tests see apache/spark#47856

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[4.1.0] Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSu…

9e2a371

…ite and GlutenJoinSuite tests see apache/spark#47856

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

Uh oh!

Conversation

cxzl25 commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

HyukjinKwon commented Aug 26, 2024

Uh oh!

dongjoon-hyun commented Sep 11, 2024

Uh oh!

mridulm commented Sep 12, 2024

Uh oh!

mridulm commented Sep 12, 2024

Uh oh!

dongjoon-hyun commented Sep 12, 2024

Uh oh!

pan3793 commented Apr 18, 2025

Uh oh!

mridulm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mridulm Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mridulm Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

mridulm Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

mridulm commented May 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rahil-c commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mridulm commented Jun 25, 2025

Uh oh!

attilapiros commented Jun 25, 2025

Uh oh!

attilapiros left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cxzl25 commented Aug 23, 2024 •

edited

Loading

rahil-c commented Jun 24, 2025 •

edited

Loading