HADOOP-17343. Upgrade AWS SDK to 1.11.901 #2468

steveloughran · 2020-11-17T14:15:11Z

Upgrade the AWS SDK to the latest release

Includes the changes to the manual steps of the regression testing needed to test directory marker support

Tested AWS London (v4 signing only, BTW)

Integration tests ( -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep -Ds3guard -Ddynamo -Dfs.s3a.directory.marker.audit=true) with review of all output files.
Downstream spark build and run through spark-cloud-integration suite
Manual commands from the testing document

steveloughran · 2020-11-17T14:16:07Z

@dongjoon-hyun @sunchao

steveloughran · 2020-11-17T14:31:48Z

Here's the cloud storage dependencies BTW

[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0-SNAPSHOT:compile
[INFO] |  +- com.amazonaws:aws-java-sdk-bundle:jar:1.11.901:compile
[INFO] |  \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile

I think we should consider downgrading that openssl dependency in the hadoop-aws pom from "runtime" to "test". It's clearly creeping in downstream and we don't need that.

hadoop-yetus · 2020-11-18T05:03:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 35s		trunk passed
+1 💚	compile	23m 55s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	18m 35s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	mvnsite	26m 11s		trunk passed
+1 💚	shadedclient	17m 26s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 26s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 39s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 39s		Maven dependency ordering for patch
+1 💚	mvninstall	24m 4s		the patch passed
+1 💚	compile	21m 12s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	21m 12s		the patch passed
+1 💚	compile	18m 34s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	18m 34s		the patch passed
+1 💚	mvnsite	22m 55s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 14s		There were no new shelldocs issues.
-1 ❌	whitespace	0m 0s	/whitespace-eol.txt	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	19m 14s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 47s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 47s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
			_ Other Tests _
-1 ❌	unit	614m 20s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 29s		The patch does not generate ASF License warnings.
		887m 9s

Reason	Tests
Failed junit tests	hadoop.mapred.gridmix.TestGridmixSubmission
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestBlobMetadata
	hadoop.fs.azure.TestWasbFsck
	hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption
	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
	hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
	hadoop.yarn.service.TestYarnNativeServices
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.hdfs.TestViewDistributedFileSystem
	hadoop.hdfs.TestDecommission
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/1/artifact/out/Dockerfile
GITHUB PR	#2468
Optional Tests	dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml markdownlint
uname	Linux 645a7afb9aaf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b57f04c`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/1/testReport/
Max. process+thread count	2786 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/1/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

dongjoon-hyun · 2020-11-18T16:13:02Z

Super! Thank you so much for taking over, @steveloughran .

steveloughran · 2020-11-19T14:18:36Z

merge conflict due to me committing something I shouldn't have. will revert

Change-Id: I72bc0e96ded326708cfde07c44667ea600f86084

hadoop-yetus · 2020-11-20T05:53:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 13s		trunk passed
+1 💚	compile	19m 56s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 26s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	mvnsite	25m 28s		trunk passed
+1 💚	shadedclient	15m 44s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 10s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 26s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 45s		Maven dependency ordering for patch
+1 💚	mvninstall	21m 22s		the patch passed
+1 💚	compile	19m 24s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 24s		the patch passed
+1 💚	compile	17m 23s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 23s		the patch passed
+1 💚	mvnsite	20m 8s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 18s		There were no new shelldocs issues.
-1 ❌	whitespace	0m 0s	/whitespace-eol.txt	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 23s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 12s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 30s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
			_ Other Tests _
-1 ❌	unit	611m 45s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	1m 52s	/patch-asflicense-problems.txt	The patch generated 2 ASF License warnings.
		858m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.server.namenode.TestFSImage
	hadoop.yarn.service.TestYarnNativeServices
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.fs.azure.TestBlobMetadata
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestWasbFsck
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/3/artifact/out/Dockerfile
GITHUB PR	#2468
Optional Tests	dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml markdownlint
uname	Linux 264ee8ff4825 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `34aa613`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/3/testReport/
Max. process+thread count	3758 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2468/3/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-11-21T14:46:03Z

test failures unrelated (wasb fix in progress, license fix for the lz4 files in).

@sunchao -want to look/approve this?

sunchao

LGTM. Checked test failures and they are not related. Thanks @steveloughran for pinging.

Contributed by Steve Loughran.

Contributed by Steve Loughran. Change-Id: I8ea1e248f9b145471118ea729a6a67e40acea6ff

Contributed by Steve Loughran Change-Id: I87d7288acdfc05cc5fcf262d9229373fd4bda652

steveloughran added build enhancement fs/s3 changes related to hadoop-aws; submitter must declare test endpoint labels Nov 17, 2020

risdenk approved these changes Nov 17, 2020

View reviewed changes

steveloughran mentioned this pull request Nov 17, 2020

HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

Closed

HADOOP-17343. Upgrade AWS SDK to 1.11.901

72b2a77

Change-Id: I72bc0e96ded326708cfde07c44667ea600f86084

steveloughran force-pushed the s3/HADOOP-17343-aws-sdk-upgrade branch from 69c8760 to 72b2a77 Compare November 19, 2020 15:32

This comment has been minimized.

Sign in to view

sunchao approved these changes Nov 21, 2020

View reviewed changes

steveloughran merged commit fb79be9 into apache:trunk Nov 23, 2020

asfgit pushed a commit that referenced this pull request Nov 23, 2020

HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468)

e4bc64c

Contributed by Steve Loughran.

steveloughran deleted the s3/HADOOP-17343-aws-sdk-upgrade branch November 23, 2020 15:45

asfgit pushed a commit that referenced this pull request Dec 3, 2020

HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468)

3efd993

Contributed by Steve Loughran. Change-Id: I8ea1e248f9b145471118ea729a6a67e40acea6ff

jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023

CDPD-19733. HADOOP-17343. Upgrade AWS SDK to 1.11.901 (apache#2468)

d9f7e32

Contributed by Steve Loughran Change-Id: I87d7288acdfc05cc5fcf262d9229373fd4bda652

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17343. Upgrade AWS SDK to 1.11.901 #2468

HADOOP-17343. Upgrade AWS SDK to 1.11.901 #2468

Uh oh!

steveloughran commented Nov 17, 2020 •

edited

Loading

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

hadoop-yetus commented Nov 18, 2020

Uh oh!

dongjoon-hyun commented Nov 18, 2020 •

edited

Loading

Uh oh!

steveloughran commented Nov 19, 2020

Uh oh!

This comment has been minimized.

hadoop-yetus commented Nov 20, 2020

Uh oh!

steveloughran commented Nov 21, 2020

Uh oh!

sunchao left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HADOOP-17343. Upgrade AWS SDK to 1.11.901 #2468

HADOOP-17343. Upgrade AWS SDK to 1.11.901 #2468

Uh oh!

Conversation

steveloughran commented Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

hadoop-yetus commented Nov 18, 2020

Uh oh!

dongjoon-hyun commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Nov 19, 2020

Uh oh!

This comment has been minimized.

hadoop-yetus commented Nov 20, 2020

Uh oh!

steveloughran commented Nov 21, 2020

Uh oh!

sunchao left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

steveloughran commented Nov 17, 2020 •

edited

Loading

dongjoon-hyun commented Nov 18, 2020 •

edited

Loading