[SPARK-28355][CORE][PYTHON] Use Spark conf for threshold at which command is compressed by broadcast #25123
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
ok to test |
gatorsmile
reviewed
Jul 11, 2019
| private[spark] val BROADCAST_UDF_THRESHOLD = ConfigBuilder("spark.broadcast.UDFThreshold") | ||
| .doc("The threshold at which a serialized command is compressed by broadcast, in " + | ||
| "bytes unless otherwise specified") | ||
| .bytesConf(ByteUnit.BYTE) |
maropu
changed the title
maropu
reviewed
Jul 11, 2019
gatorsmile
reviewed
Jul 11, 2019
| "mechanisms to guarantee data won't be corrupted during broadcast") | ||
| .booleanConf.createWithDefault(true) | ||
|
|
||
| private[spark] val BROADCAST_UDF_THRESHOLD = ConfigBuilder("spark.broadcast.UDFThreshold") |
Member
There was a problem hiding this comment.
The variable name looks confusing. COMMAND_COMPRESSION_THRESHOLD?
Member
|
cc @HyukjinKwon |
|
Test build #107560 has finished for PR 25123 at commit
|
HyukjinKwon
reviewed
Jul 12, 2019
|
|
||
| private[spark] val BROADCAST_FOR_UDF_COMPRESSION_THRESHOLD = | ||
| ConfigBuilder("spark.broadcast.UDFCompressionThreshold") | ||
| .doc("The threshold at which a a user-defined function (UDF) is compressed by broadcast, " + |
Member
There was a problem hiding this comment.
I think this also applies to RDD APIs. We can just say, for instance, The threshold at which Python commands for RDD APIs and user-defined function (UDF) are serialized by broadcast .... Feel free to change wording.
Contributor
Author
There was a problem hiding this comment.
I updated the description to include this.
|
Test build #107564 has finished for PR 25123 at commit
|
|
Test build #107603 has finished for PR 25123 at commit
|
|
Test build #107623 has finished for PR 25123 at commit
|
Member
|
LGTM Thanks! Merged to master. |
Member
|
LGTM too |
vinodkc
pushed a commit
to vinodkc/spark
that referenced
this pull request
Jul 18, 2019
…mand is compressed by broadcast ## What changes were proposed in this pull request? The `_prepare_for_python_RDD` method currently broadcasts a pickled command if its length is greater than the hardcoded value `1 << 20` (1M). This change sets this value as a Spark conf instead. ## How was this patch tested? Unit tests, manual tests. Closes apache#25123 from jessecai/SPARK-28355. Authored-by: Jesse Cai <[email protected]> Signed-off-by: gatorsmile <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The
_prepare_for_python_RDDmethod currently broadcasts a pickled command if its length is greater than the hardcoded value1 << 20(1M). This change sets this value as a Spark conf instead.How was this patch tested?
Unit tests, manual tests.