YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor #3325

akshatb1 · 2021-08-24T06:36:50Z

Description of PR

Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor

How was this patch tested?

Added tests for RouterMetrics and validated existing test cases for FederationClientInterceptor APIs. Unit test are running successfully on dev machine.
Verified on a test cluster.

…rationClientInterceptor

hadoop-yetus · 2021-08-24T10:03:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 51s		trunk passed
+1 💚	compile	23m 2s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 27s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 55s		trunk passed
+1 💚	mvnsite	2m 10s		trunk passed
+1 💚	javadoc	1m 34s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 6s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 16s		trunk passed
+1 💚	shadedclient	17m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	22m 0s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 0s		the patch passed
+1 💚	compile	19m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 49s		the patch passed
+1 💚	mvnsite	2m 7s		the patch passed
+1 💚	javadoc	1m 34s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 6s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	1m 4s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	16m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 46s		hadoop-common in the patch passed.
+1 💚	unit	2m 39s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		205m 47s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
	Dead store to applications in org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getApplications(GetApplicationsRequest) At FederationClientInterceptor.java:org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getApplications(GetApplicationsRequest) At FederationClientInterceptor.java:[line 642]
	Dead store to clusterMetrics in org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getClusterMetrics(GetClusterMetricsRequest) At FederationClientInterceptor.java:org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getClusterMetrics(GetClusterMetricsRequest) At FederationClientInterceptor.java:[line 669]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/1/artifact/out/Dockerfile
GITHUB PR	#3325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell spotbugs checkstyle
uname	Linux 7d8c0eccad9e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `84c56b0`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/1/testReport/
Max. process+thread count	3137 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-08-24T20:45:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 44s		trunk passed
+1 💚	compile	25m 1s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	22m 9s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 50s		trunk passed
+1 💚	mvnsite	2m 10s		trunk passed
+1 💚	javadoc	1m 35s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 8s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 13s		trunk passed
+1 💚	shadedclient	16m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	22m 18s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 18s		the patch passed
+1 💚	compile	20m 44s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	20m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 5s	/results-checkstyle-root.txt	root: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	2m 4s		the patch passed
+1 💚	javadoc	1m 32s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 7s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 51s		the patch passed
+1 💚	shadedclient	18m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 3s		hadoop-common in the patch passed.
+1 💚	unit	2m 42s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		216m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/2/artifact/out/Dockerfile
GITHUB PR	#3325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell spotbugs checkstyle
uname	Linux 364ae1cd4e0b 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5ce2ccf`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/2/testReport/
Max. process+thread count	1361 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

akshatb1 · 2021-09-01T16:02:27Z

@goiri @bibinchundatt Could you kindly help in reviewing this PR? Thanks.

...src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java

...yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouterMetrics.java

hadoop-yetus · 2021-09-07T09:26:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 3s		trunk passed
+1 💚	compile	28m 26s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	23m 30s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	4m 5s		trunk passed
+1 💚	mvnsite	2m 31s		trunk passed
+1 💚	javadoc	1m 43s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 16s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 39s		trunk passed
+1 💚	shadedclient	17m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 33s		the patch passed
+1 💚	compile	24m 1s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	24m 1s		the patch passed
+1 💚	compile	23m 5s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	23m 5s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 1s	/results-checkstyle-root.txt	root: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	2m 19s		the patch passed
+1 💚	javadoc	2m 4s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 11s		the patch passed
+1 💚	shadedclient	17m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 6s		hadoop-common in the patch passed.
+1 💚	unit	2m 47s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		229m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/3/artifact/out/Dockerfile
GITHUB PR	#3325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell spotbugs checkstyle
uname	Linux 1d2f0ae22317 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `320c37d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/3/testReport/
Max. process+thread count	1816 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-09-07T17:15:54Z

...src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java

 import java.lang.reflect.Method;
 import java.util.ArrayList;
 import java.util.Collection;
+import java.util.HashMap;


There is a warning here.

hadoop-yetus · 2021-09-08T07:54:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 49s		trunk passed
+1 💚	compile	23m 6s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 47s		trunk passed
+1 💚	mvnsite	2m 9s		trunk passed
+1 💚	javadoc	1m 35s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 8s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	16m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	22m 15s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 15s		the patch passed
+1 💚	compile	19m 45s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 49s		the patch passed
+1 💚	mvnsite	2m 6s		the patch passed
+1 💚	javadoc	1m 34s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 3s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 42s		the patch passed
+1 💚	shadedclient	17m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 22s		hadoop-common in the patch passed.
+1 💚	unit	2m 38s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		207m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/4/artifact/out/Dockerfile
GITHUB PR	#3325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell spotbugs checkstyle
uname	Linux 1860d29c9667 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7c30013`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/4/testReport/
Max. process+thread count	2289 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3325/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

…APIs in FederationClientInterceptor (apache#3325)

akshatb1 added 2 commits August 24, 2021 11:54

Adding metrics for getClusterMetrics and getApplications APIs in Fede…

8c9824b

…rationClientInterceptor

Removing the inadvertent log files

84c56b0

Removing redundant initialization

5ce2ccf

goiri reviewed Sep 1, 2021

View reviewed changes

...src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java Show resolved Hide resolved

...yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouterMetrics.java Outdated Show resolved Hide resolved

Applying Hadoop CodeStyle

320c37d

goiri reviewed Sep 7, 2021

View reviewed changes

Removing unused import

7c30013

goiri approved these changes Sep 8, 2021

View reviewed changes

goiri merged commit dee6dc2 into apache:trunk Sep 9, 2021

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

YARN-10893. Adding metrics for getClusterMetrics and getApplications …

6750af5

…APIs in FederationClientInterceptor (apache#3325)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor #3325

YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor #3325

Uh oh!

akshatb1 commented Aug 24, 2021

Uh oh!

hadoop-yetus commented Aug 24, 2021

Uh oh!

hadoop-yetus commented Aug 24, 2021

Uh oh!

akshatb1 commented Sep 1, 2021

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Sep 7, 2021

Uh oh!

goiri Sep 7, 2021

Uh oh!

akshatb1 Sep 8, 2021

Uh oh!

hadoop-yetus commented Sep 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor #3325

YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor #3325

Uh oh!

Conversation

akshatb1 commented Aug 24, 2021

Description of PR

How was this patch tested?

Uh oh!

hadoop-yetus commented Aug 24, 2021

Uh oh!

hadoop-yetus commented Aug 24, 2021

Uh oh!

akshatb1 commented Sep 1, 2021

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Sep 7, 2021

Uh oh!

goiri Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

akshatb1 Sep 8, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants