HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. #4704

mukund-thakur · 2022-08-04T20:27:17Z

Description of PR

Propagate vectored s3a input stream stats to file system stats.

How was this patch tested?

Re-ran the vectored io tests using us-east-1 endpoint.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…m stats.

hadoop-yetus · 2022-08-05T00:31:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 24s		trunk passed
+1 💚	compile	23m 14s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 2s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 24s		trunk passed
+1 💚	mvnsite	3m 47s		trunk passed
+1 💚	javadoc	2m 54s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 3s		trunk passed
+1 💚	shadedclient	22m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 45s		the patch passed
+1 💚	compile	22m 32s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 32s		the patch passed
+1 💚	compile	21m 1s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 22s		the patch passed
+1 💚	mvnsite	3m 37s		the patch passed
+1 💚	javadoc	2m 50s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 12s		the patch passed
+1 💚	shadedclient	22m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 50s		hadoop-common in the patch passed.
+1 💚	unit	3m 34s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 35s		The patch does not generate ASF License warnings.
		243m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/1/artifact/out/Dockerfile
GITHUB PR	#4704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7114ac705ada 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fef7f5f`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/1/testReport/
Max. process+thread count	1330 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

+1

steveloughran · 2022-08-08T10:25:57Z

...oop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java

-          LOG.error("Exception while running vectored read ", ex);
-          Assert.fail("Exception while running vectored read " + ex);
-        }
+        intercept(ExecutionException.class, (Callable<ByteBuffer>) data::get);


there's an interceptFuture() variant where you can also specify the timeout

hadoop-yetus · 2022-08-10T23:05:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 36s		trunk passed
+1 💚	compile	23m 20s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 45s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 22s		trunk passed
+1 💚	mvnsite	3m 35s		trunk passed
+1 💚	javadoc	2m 57s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 5s		trunk passed
+1 💚	shadedclient	22m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 47s		the patch passed
+1 💚	compile	22m 28s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 28s		the patch passed
+1 💚	compile	20m 43s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 9s	/results-checkstyle-root.txt	root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	3m 44s		the patch passed
+1 💚	javadoc	2m 48s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 11s		the patch passed
+1 💚	shadedclient	22m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	18m 55s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	unit	3m 25s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 37s		The patch does not generate ASF License warnings.
		242m 36s

Reason	Tests
Failed junit tests	hadoop.fs.contract.localfs.TestLocalFSContractVectoredRead
	hadoop.fs.contract.rawlocal.TestRawLocalContractVectoredRead

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/2/artifact/out/Dockerfile
GITHUB PR	#4704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e9a142c077c3 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a65a68d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/2/testReport/
Max. process+thread count	1859 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-08-11T04:01:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 25s		trunk passed
+1 💚	compile	23m 13s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 50s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 23s		trunk passed
+1 💚	mvnsite	3m 47s		trunk passed
+1 💚	javadoc	2m 50s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 36s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 1s		trunk passed
+1 💚	shadedclient	22m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 42s		the patch passed
+1 💚	compile	22m 29s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 29s		the patch passed
+1 💚	compile	20m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 11s		the patch passed
+1 💚	mvnsite	3m 36s		the patch passed
+1 💚	javadoc	2m 51s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 40s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 16s		the patch passed
+1 💚	shadedclient	22m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 48s		hadoop-common in the patch passed.
+1 💚	unit	3m 22s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 36s		The patch does not generate ASF License warnings.
		242m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/3/artifact/out/Dockerfile
GITHUB PR	#4704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c9c07ee729e1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cad8559`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/3/testReport/
Max. process+thread count	3159 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4704/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

+1

…m stats. (#4704) part of HADOOP-18103. Contributed By: Mukund Thakur

…m stats. (apache#4704) part of HADOOP-18103. Contributed By: Mukund Thakur

HADOOP-18392. Propagate vectored s3a input stream stats to file syste…

fef7f5f

…m stats.

mukund-thakur requested a review from steveloughran August 5, 2022 15:55

steveloughran approved these changes Aug 8, 2022

View reviewed changes

using interceptFuture with timeout

a65a68d

checkstyle

cad8559

steveloughran approved these changes Aug 11, 2022

View reviewed changes

mukund-thakur merged commit b28e4c6 into apache:trunk Aug 11, 2022

mukund-thakur added a commit that referenced this pull request Aug 11, 2022

HADOOP-18392. Propagate vectored s3a input stream stats to file syste…

93c4704

…m stats. (#4704) part of HADOOP-18103. Contributed By: Mukund Thakur

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18392. Propagate vectored s3a input stream stats to file syste…

0e09557

…m stats. (apache#4704) part of HADOOP-18103. Contributed By: Mukund Thakur

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. #4704

HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. #4704

Uh oh!

mukund-thakur commented Aug 4, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented Aug 5, 2022

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Aug 8, 2022

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

hadoop-yetus commented Aug 11, 2022

Uh oh!

steveloughran left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. #4704

HADOOP-18392. Propagate vectored s3a input stream stats to file system stats. #4704

Uh oh!

Conversation

mukund-thakur commented Aug 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Aug 5, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Aug 8, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Aug 10, 2022

Uh oh!

hadoop-yetus commented Aug 11, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mukund-thakur commented Aug 4, 2022 •

edited

Loading