HADOOP-19097. S3A: Set fs.s3a.connection.establish.timeout to 30s #6601
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is consistent with the java value. Change-Id: Ib24f4057f778206d6d59230de02037c5ada209f4
|
💔 -1 overall
This message was automatically generated. |
|
@HarshitGupta11 @mukund-thakur @ahmarsuhail reviews, please? targeting 3.4.1 |
|
Change looks good, until we remove fs.s3a settings from core-default, i wonder if we should do one round of comparison b/ s3a/Contants and core-default (as a followup jira)? |
ayushtkn
approved these changes
Mar 4, 2024
|
@virajjasani I did a scan. Now, the clever thing would be to have a test suite which compared values so there is never a regression. e.g assertDurationEqual(conf, CONNECTION_TIMEOUT, CONNECTION_TIMEOUT_DEFAULT) |
dongjoon-hyun
added a commit
to apache/spark
that referenced
this pull request
Mar 25, 2024
…eout` to 30s if missing ### What changes were proposed in this pull request? This PR aims to handle HADOOP-19097 from Apache Spark side. We can remove this when Apache Hadoop `3.4.1` releases. - apache/hadoop#6601 ### Why are the changes needed? Apache Hadoop shows a warning to its default configuration. This default value issue is fixed at Apache Spark 3.4.1. ``` 24/03/25 14:46:21 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead ``` This change will suppress Apache Hadoop default warning in the consistent way with the future Hadoop releases. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Manually. **BUILD** ``` $ dev/make-distribution.sh -Phadoop-cloud ``` **BEFORE** ``` scala> spark.range(10).write.mode("overwrite").orc("s3a://express-1-zone--***--x-s3/orc/") ... 24/03/25 15:50:46 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead ``` **AFTER** ``` scala> spark.range(10).write.mode("overwrite").orc("s3a://express-1-zone--***--x-s3/orc/") ...(ConfigurationHelper warning is gone)... ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45710 from dongjoon-hyun/SPARK-47552. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
sweisdb
pushed a commit
to sweisdb/spark
that referenced
this pull request
Apr 1, 2024
…eout` to 30s if missing ### What changes were proposed in this pull request? This PR aims to handle HADOOP-19097 from Apache Spark side. We can remove this when Apache Hadoop `3.4.1` releases. - apache/hadoop#6601 ### Why are the changes needed? Apache Hadoop shows a warning to its default configuration. This default value issue is fixed at Apache Spark 3.4.1. ``` 24/03/25 14:46:21 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead ``` This change will suppress Apache Hadoop default warning in the consistent way with the future Hadoop releases. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Manually. **BUILD** ``` $ dev/make-distribution.sh -Phadoop-cloud ``` **BEFORE** ``` scala> spark.range(10).write.mode("overwrite").orc("s3a://express-1-zone--***--x-s3/orc/") ... 24/03/25 15:50:46 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead ``` **AFTER** ``` scala> spark.range(10).write.mode("overwrite").orc("s3a://express-1-zone--***--x-s3/orc/") ...(ConfigurationHelper warning is gone)... ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45710 from dongjoon-hyun/SPARK-47552. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is consistent with the java value.
I had thought of cutting all the fs.s3a settings from core-default, but think we maybe need to review our public docs before doing that. having to look at Constants.java shouldn't be the default way to learn about an option.
How was this patch tested?
commented out my timeout from my auth-keys file (so it wasn't stamping on this default) and running the tests. The used ripgrep to look for the "is too low" message; only found in test cases where we explicitly created the problem.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?