Skip to content

[ABFS] [Backport 3.4] Back Merging PRs from trunk to Branch 3.4 #6611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[ABFS] [Backport 3.4] Back Merging PRs from trunk to Branch 3.4 #6611

wants to merge 3 commits into from

Conversation

anujmodi2021
Copy link
Contributor

@anujmodi2021 anujmodi2021 commented Mar 5, 2024 •
edited
Loading

Description of PR

Following two PRs were merged to trunk.
#5881
#6069

These two PRs may have conflicts thus creating this single backport PR to avoid any merge issues.
Commits Backported from trunk:
1336c36
99b9e7f

How was this patch tested?

Both the commits were cherry-picked and ABFS Test Suite was run.

@anujmodi2021
Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy…
395159a 
… for connection timeout. (apache#5881)


Contributed By: Anuj Modi
@anujmodi2021
HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity veri…
c9cd613 
…fication of the request content during transport (apache#6069)


Contributed By: Anuj Modi
@anujmodi2021 anujmodi2021 changed the base branch from trunk to branch-3.4 March 5, 2024 05:34
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 23s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 18 new or modified test files.
_ branch-3.4 Compile Tests _
+1 💚 mvninstall 32m 38s branch-3.4 passed
+1 💚 compile 0m 19s branch-3.4 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 0m 20s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 22s branch-3.4 passed
+1 💚 mvnsite 0m 28s branch-3.4 passed
+1 💚 javadoc 0m 26s branch-3.4 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 22s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 45s branch-3.4 passed
+1 💚 shadedclient 19m 37s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 19s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 14s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 14s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 13s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
+1 💚 mvnsite 0m 18s the patch passed
+1 💚 javadoc 0m 18s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 18s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 41s the patch passed
+1 💚 shadedclient 19m 16s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 1m 51s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 24s The patch does not generate ASF License warnings.
82m 19s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/artifact/out/Dockerfile
GITHUB PR #6611
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname Linux b238ccd57596 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / c9cd613
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/testReport/
Max. process+thread count 551 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@anujmodi2021 anujmodi2021 marked this pull request as ready for review March 6, 2024 05:01
@anujmodi2021
Copy link
Contributor Author

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 9
[INFO] Results:
[INFO]
[WARNING] Tests run: 604, Failures: 0, Errors: 0, Skipped: 268
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

Time taken: 22 mins 43 secs.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 18 new or modified test files.
_ branch-3.4 Compile Tests _
+1 💚 mvninstall 31m 55s branch-3.4 passed
+1 💚 compile 0m 24s branch-3.4 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 0m 20s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 21s branch-3.4 passed
+1 💚 mvnsite 0m 28s branch-3.4 passed
+1 💚 javadoc 0m 27s branch-3.4 passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 24s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 43s branch-3.4 passed
+1 💚 shadedclient 19m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 20s the patch passed
+1 💚 compile 0m 18s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 0m 18s the patch passed
+1 💚 compile 0m 16s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 13s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10)
+1 💚 mvnsite 0m 17s the patch passed
+1 💚 javadoc 0m 17s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 40s the patch passed
+1 💚 shadedclient 19m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 1m 51s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
88m 45s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/2/artifact/out/Dockerfile
GITHUB PR #6611
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname Linux d73fbc845651 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / 94fc618
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/2/testReport/
Max. process+thread count 552 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6611/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

This was referenced Mar 6, 2024
Merged
Merged
@steveloughran
Copy link
Contributor

I think what I will do here is check this branch out then cherrypick each PR in order, so that they are still two separate PRs. Do you see any problem with this?

@mukund-thakur
Copy link
Contributor

I think it will depend on the fact first commit works as it its and is not dependent on second one. We need to merge these separately such that if we have to revert in future we can revert only one.
@anujmodi2021 there shouldn't be any problem with that right?

@mukund-thakur
Copy link
Contributor

Cp'ed the first one, compiled and ran azure tests. It went fine. Will do the same for 2nd one and combine the checkstyle in 2nd one.

@anujmodi2021
Copy link
Contributor Author

I think it will depend on the fact first commit works as it its and is not dependent on second one. We need to merge these separately such that if we have to revert in future we can revert only one. @anujmodi2021 there shouldn't be any problem with that right?

Right..
This shouldn't be a problem. Both the changes are independent of each other. Its just that some commons files are modified.
If there is a need to revert any one of them, then we might have to resolve some merge conflicts. Nothing should break in production.

@mukund-thakur
Copy link
Contributor

Merged into branch-3.4.

@anujmodi2021 anujmodi2021 deleted the branch-3.4-dev branch March 14, 2024 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ABFS AWS build Common DistCp DYNAMOMETER HDFS Infra MapReduce Maven-Plugins Native RBF TOOLS trunk YARN
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@anujmodi2021 @hadoop-yetus @steveloughran @mukund-thakur