MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb #5014

hotcodemacha · 2022-10-13T12:06:32Z

Description of PR

Replace and deprecate mapred.tasktracker.indexcache.mb

In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size of the shuffle service index cache. As the tasktracker no longer exists, we should replace this with something like mapreduce.reduce.shuffle.indexcache.mb.

How was this patch tested?

UT

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-10-13T16:41:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	29m 35s		trunk passed
+1 💚	compile	27m 50s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	23m 47s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 45s		trunk passed
+1 💚	mvnsite	3m 28s		trunk passed
+1 💚	javadoc	2m 30s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 7s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 21s		trunk passed
+1 💚	shadedclient	25m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 50s		the patch passed
+1 💚	compile	27m 24s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	27m 24s		the patch passed
+1 💚	compile	23m 37s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	23m 37s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 37s	/results-checkstyle-root.txt	root: The patch generated 1 new + 671 unchanged - 1 fixed = 672 total (was 672)
+1 💚	mvnsite	3m 29s		the patch passed
+1 💚	javadoc	2m 28s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 2s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 36s		the patch passed
+1 💚	shadedclient	26m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 26s		hadoop-common in the patch passed.
+1 💚	unit	7m 41s		hadoop-mapreduce-client-core in the patch passed.
+1 💚	asflicense	1m 22s		The patch does not generate ASF License warnings.
		271m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/1/artifact/out/Dockerfile
GITHUB PR	#5014
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 9da8043d20c6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2cec8a3`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/1/testReport/
Max. process+thread count	2991 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-10-13T21:17:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 0s		Maven dependency ordering for branch
+1 💚	mvninstall	29m 57s		trunk passed
+1 💚	compile	27m 51s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	23m 29s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 39s		trunk passed
+1 💚	mvnsite	3m 28s		trunk passed
+1 💚	javadoc	2m 38s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 5s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 21s		trunk passed
+1 💚	shadedclient	26m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 57s		the patch passed
+1 💚	compile	26m 39s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	26m 39s		the patch passed
+1 💚	compile	23m 27s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	23m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 34s		root: The patch generated 0 new + 671 unchanged - 1 fixed = 671 total (was 672)
+1 💚	mvnsite	3m 27s		the patch passed
+1 💚	javadoc	2m 25s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 4s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 32s		the patch passed
+1 💚	shadedclient	25m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 23s		hadoop-common in the patch passed.
+1 💚	unit	7m 45s		hadoop-mapreduce-client-core in the patch passed.
+1 💚	asflicense	1m 19s		The patch does not generate ASF License warnings.
		270m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/2/artifact/out/Dockerfile
GITHUB PR	#5014
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 5beb15af47f9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9cec920`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/2/testReport/
Max. process+thread count	3144 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hotcodemacha · 2022-10-15T11:43:45Z

@aajisaka @goiri @ZanderXu - Please help in reviewing it. Thank you so much.

ZanderXu · 2022-10-17T03:06:39Z

@ashutoshcipher Thanks for involving me here. I just focus on HDFS right now, so I can't give any review ideas here. Sorry.

hotcodemacha · 2022-10-17T09:37:13Z

Thanks @ZanderXu. Will ask you for help in HDFS jiras :)

hotcodemacha · 2022-11-02T01:46:03Z

Hi @cnauroth, Please help in reviewing this PR. Thanks

...ient/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java

aajisaka

Thank you @ashutoshcipher for the cleanup. I have 1 comment

hotcodemacha · 2022-11-08T20:16:32Z

Thank you @ashutoshcipher for the cleanup. I have 1 comment

Thanks @aajisaka for reviewing. I have addressed your comment in my last commit.

hadoop-yetus · 2022-11-09T00:20:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 56s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 6s		trunk passed
+1 💚	compile	23m 30s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	20m 51s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 15s		trunk passed
+1 💚	mvnsite	3m 12s		trunk passed
+1 💚	javadoc	2m 42s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 59s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 16s		trunk passed
+1 💚	shadedclient	22m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 44s		the patch passed
+1 💚	compile	22m 32s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	22m 32s		the patch passed
+1 💚	compile	20m 58s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 58s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 14s	/results-checkstyle-root.txt	root: The patch generated 1 new + 671 unchanged - 1 fixed = 672 total (was 672)
+1 💚	mvnsite	3m 11s		the patch passed
+1 💚	javadoc	2m 34s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 10s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 54s		the patch passed
+1 💚	shadedclient	22m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 56s		hadoop-common in the patch passed.
+1 💚	unit	7m 29s		hadoop-mapreduce-client-core in the patch passed.
+1 💚	asflicense	1m 16s		The patch does not generate ASF License warnings.
		243m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/3/artifact/out/Dockerfile
GITHUB PR	#5014
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux a834873ee1b6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6131706`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/3/testReport/
Max. process+thread count	3071 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5014/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hotcodemacha · 2022-11-09T14:21:45Z

Checkstyle in latest Yetus can be ignored as public static final is added to keep it consistent with other parameters.

hotcodemacha · 2022-11-11T04:15:40Z

@aajisaka Addressed your comments, please help in reviewing it. Thanks.

aajisaka

+1

aajisaka · 2022-11-14T02:08:13Z

Thank you @ashutoshcipher for taking this issue over.

apache#5014) Co-authored-by: Ashutosh Gupta <[email protected]> Signed-off-by: Akira Ajisaka <[email protected]>

MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb

2cec8a3

hotcodemacha marked this pull request as draft October 13, 2022 12:06

remove unused import

9cec920

hotcodemacha marked this pull request as ready for review October 13, 2022 22:06

aajisaka reviewed Nov 8, 2022

View reviewed changes

...ient/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java Outdated Show resolved Hide resolved

aajisaka requested changes Nov 8, 2022

View reviewed changes

making SHUFFLE_INDEX_CACHE public static final

6131706

hotcodemacha requested a review from aajisaka November 9, 2022 14:20

aajisaka approved these changes Nov 14, 2022

View reviewed changes

aajisaka merged commit a48e8c9 into apache:trunk Nov 14, 2022

MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb #5014

MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb #5014

Uh oh!

Conversation

hotcodemacha commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Oct 13, 2022

Uh oh!

hadoop-yetus commented Oct 13, 2022

Uh oh!

hotcodemacha commented Oct 15, 2022

Uh oh!

ZanderXu commented Oct 17, 2022

Uh oh!

hotcodemacha commented Oct 17, 2022

Uh oh!

hotcodemacha commented Nov 2, 2022

Uh oh!

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

hotcodemacha commented Nov 8, 2022

Uh oh!

hadoop-yetus commented Nov 9, 2022

Uh oh!

hotcodemacha commented Nov 9, 2022

Uh oh!

hotcodemacha commented Nov 11, 2022

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

aajisaka commented Nov 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hotcodemacha commented Oct 13, 2022 •

edited

Loading