HADOOP-17451. IOStatistics test failures in S3A code. #2594

steveloughran · 2021-01-05T20:55:51Z

Fixing tests which fail intermittently based on configs and
in the case of the HugeFile tests, only in bulk runs when existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Fixes:

HADOOP-17451. HugeFile upload tests
HADOOP-17456. ITestPartialRenamesDeletes.testPartialDirDelete failure
HADOOP-17455. ITestS3ADeleteCost failure

Testing: s3 london with -Dparallel-tests -DtestsThreadCount=6 -Dscale -Dmarkers=keep -Ds3guard -Ddynamo

no test failures!

steveloughran · 2021-01-06T11:52:01Z

yetus failure from OOM/thread issues.

steveloughran · 2021-01-07T14:17:37Z

style

./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java:702:      bulkDeleteVerbCount.assertDiffEquals("Wrong count of delete operations in ": Line is longer than 80 characters (found 81). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java:29:import org.apache.hadoop.fs.statistics.IOStatistics;:8: Unused import - org.apache.hadoop.fs.statistics.IOStatistics. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java:30:import org.apache.hadoop.metrics2.lib.MutableGaugeLong;:8: Unused import - org.apache.hadoop.metrics2.lib.MutableGaugeLong. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java:32:import org.assertj.core.api.Assertions;:8: Unused import - org.assertj.core.api.Assertions. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java:40:import static org.apache.hadoop.fs.statistics.IOStatisticsLogging.ioStatisticsToString;:15: Unused import - org.apache.hadoop.fs.statistics.IOStatisticsLogging.ioStatisticsToString. [UnusedImports]

steveloughran · 2021-01-08T18:17:53Z

-checkstyle of unused import; fixing.

Ready for review: @bgaborg @mukund-thakur @mehakmeet @sunchao

Fixing tests which fail intermittently based on configs and in the case of the HugeFile tests, only in bulk runs when existing FS instances meant statistic probes sometimes ended up probing those of a previous FS. Fixes: * HADOOP-17451. HugeFile upload tests * HADOOP-17456. ITestPartialRenamesDeletes.testPartialDirDelete failure Does not fix: * HADOOP-17455. ITestS3ADeleteCost failure Change-Id: Ie6c392179a137e693ed078b4bf4932b20deea9c4

…ngleFileInDir Change-Id: I4453c54e8fbd05492c784dbf3a3dbe1238ec1025

Change-Id: I4ca26d507122b8254ddcbe56d07caced529dd665

Change-Id: Ia0ceec8aa7afc38ee5a23e7d2db4a04e9719b42b

steveloughran · 2021-01-11T13:30:39Z

last yetus run had execution problems. Rebase to trunk and resubmitting

hadoop-yetus · 2021-01-11T14:35:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	6m 48s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 32s		trunk passed
+1 💚	compile	20m 15s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 27s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 39s		trunk passed
+1 💚	mvnsite	2m 26s		trunk passed
+1 💚	shadedclient	20m 57s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 38s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 18s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 17s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 33s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 28s		the patch passed
+1 💚	compile	19m 42s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 42s		the patch passed
+1 💚	compile	17m 31s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 31s		the patch passed
+1 💚	checkstyle	2m 34s		the patch passed
+1 💚	mvnsite	2m 27s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 22s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 40s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	2m 20s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 52s		the patch passed
			_ Other Tests _
+1 💚	unit	9m 45s		hadoop-common in the patch passed.
+1 💚	unit	1m 43s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		185m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/5/artifact/out/Dockerfile
GITHUB PR	#2594
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux d7f61fc83c94 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ec22850`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/5/testReport/
Max. process+thread count	1367 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2594/5/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2021-01-11T14:42:17Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java

    public void bytesTransferred(long byteCount) {
      bytesUploaded.addAndGet(byteCount);
-      incrementGauge(STREAM_WRITE_BLOCK_UPLOADS_BYTES_PENDING, -byteCount);
+      incAllGauges(STREAM_WRITE_BLOCK_UPLOADS_BYTES_PENDING, -byteCount);


nit: why change in name as still only one guage is being incremented in the method underneath.

incAllGauges does both the iostats gauge and the Hadoop metrics gauges; we need that

mukund-thakur

LGTM +1

mehakmeet

+1, tested using mvn clean verify -Ds3guard -Ddynamo -Dauth -Dscale -Dparallel-tests on ap-south, didn't see any test failures regarding IOStats.

bgaborg · 2021-01-12T11:22:08Z

+1 if the nit is fixed which Mukund pointed out: the method naming
tests ran against Ireland without errors. Thanks for fixing it Steve!

steveloughran · 2021-01-12T15:53:35Z

Thanks! Will merge after a rebase on top of mehakmeet's

This reverts commit d3014e0. (fixing commit text before it is frozen)

Caused by HADOOP-16830 and HADOOP-17271. Fixes tests which fail intermittently based on configs and in the case of the HugeFile tests, bulk runs with existing FS instances meant statistic probes sometimes ended up probing those of a previous FS. Contributed by Steve Loughran. Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d

…ache#2594) Caused by HADOOP-16830 and HADOOP-17271. Fixes tests which fail intermittently based on configs and in the case of the HugeFile tests, bulk runs with existing FS instances meant statistic probes sometimes ended up probing those of a previous FS. Contributed by Steve Loughran. Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d

steveloughran marked this pull request as draft January 5, 2021 20:55

steveloughran added bug fs/s3 changes related to hadoop-aws; submitter must declare test endpoint test test code changes work in progress PRs still Work in Progress; reviews not expected but still welcome labels Jan 5, 2021

This comment has been minimized.

Sign in to view

steveloughran mentioned this pull request Jan 6, 2021

HADOOP-16080. hadoop-aws does not work with hadoop-client-api #2575

Merged

This comment has been minimized.

Sign in to view

steveloughran removed the work in progress PRs still Work in Progress; reviews not expected but still welcome label Jan 8, 2021

steveloughran marked this pull request as ready for review January 8, 2021 18:17

This comment has been minimized.

Sign in to view

steveloughran added 4 commits January 11, 2021 11:25

HADOOP-17455. Intermittent failure of ITestS3ADeleteCost.testDeleteSi…

3e883bf

…ngleFileInDir Change-Id: I4453c54e8fbd05492c784dbf3a3dbe1238ec1025

HADOOP-17451. IOStatistics test failures: checkstyle

5e0ae4b

Change-Id: I4ca26d507122b8254ddcbe56d07caced529dd665

HADOOP-17451. checkstyle

f303ad4

Change-Id: Ia0ceec8aa7afc38ee5a23e7d2db4a04e9719b42b

steveloughran force-pushed the s3/HADOOP-17451-iostats-test-failures branch from 16513c3 to f303ad4 Compare January 11, 2021 11:29

mukund-thakur approved these changes Jan 11, 2021

View reviewed changes

mehakmeet reviewed Jan 12, 2021

View reviewed changes

bgaborg self-requested a review January 12, 2021 08:49

bgaborg approved these changes Jan 12, 2021

View reviewed changes

steveloughran merged commit d3014e0 into apache:trunk Jan 12, 2021

asfgit pushed a commit that referenced this pull request Jan 12, 2021

Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)"

05c9c2e

This reverts commit d3014e0. (fixing commit text before it is frozen)

steveloughran mentioned this pull request Jan 12, 2021

Backport HADOOP-16830 to branch-3.3 #2616

Closed

steveloughran deleted the s3/HADOOP-17451-iostats-test-failures branch January 14, 2021 19:12

HADOOP-17451. IOStatistics test failures in S3A code. #2594

HADOOP-17451. IOStatistics test failures in S3A code. #2594

Uh oh!

Conversation

steveloughran commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

steveloughran commented Jan 6, 2021

Uh oh!

This comment has been minimized.

steveloughran commented Jan 7, 2021

Uh oh!

This comment has been minimized.

steveloughran commented Jan 8, 2021

Uh oh!

This comment has been minimized.

steveloughran commented Jan 11, 2021

Uh oh!

hadoop-yetus commented Jan 11, 2021

Uh oh!

mukund-thakur Jan 11, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Jan 12, 2021

Choose a reason for hiding this comment

Uh oh!

mukund-thakur left a comment

Choose a reason for hiding this comment

Uh oh!

mehakmeet left a comment

Choose a reason for hiding this comment

Uh oh!

bgaborg commented Jan 12, 2021

Uh oh!

steveloughran commented Jan 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

steveloughran commented Jan 5, 2021 •

edited

Loading