YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars #2075

vinayakumarb · 2020-06-15T05:28:54Z

No description provided.

hadoop-yetus · 2020-06-15T06:31:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 1s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 44s	trunk passed
+1 💚	compile	0m 24s	trunk passed
+1 💚	mvnsite	0m 48s	trunk passed
+1 💚	shadedclient	35m 3s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 16s	the patch passed
+1 💚	compile	0m 15s	the patch passed
+1 💚	javac	0m 15s	the patch passed
+1 💚	mvnsite	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	13m 34s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 18s	hadoop-client-runtime in the patch passed.
+1 💚	unit	0m 18s	hadoop-client-minicluster in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		61m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/1/artifact/out/Dockerfile
GITHUB PR	#2075
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 591f59e2a4f7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `81d8a88`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/1/testReport/
Max. process+thread count	460 (vs. ulimit of 5500)
modules	C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/1/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

vinayakumarb · 2020-06-15T09:21:33Z

@aajisaka @steveloughran @jojochuang @ayushtkn
Please review, thanks

aajisaka · 2020-06-16T08:02:50Z

hadoop-client-modules/hadoop-client-runtime/pom.xml

Agreed that jetty-client and jetty-http are shaded as well because jetty websocket-client depends on them.

websocket-client depends on jetty-xml, too. Should we shade jetty-xml or not?
https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-websocket/websocket-client/pom.xml#L25

jetty-servlet and jetty-security are brought by YARN-8778. jetty-client, jetty-http and jetty-xml are brought by HADOOP-16152. The patch of HADOOP-16152 looks excluding them on purpose.

I believe jetty-servlet and jetty-security are brought in by YARN-8763 not YARN-8778. ` ``

websocket-client depends on jetty-xml, too. Should we shade jetty-xml or not?

I think its fair to include jetty-xml as well in the shaded runtime jar. Will update the PR.

iwasakims · 2020-06-16T10:34:47Z

I ran following on hadoop dist built by mvn clean install -DskipTests -Pdist -Pnative. The patch fixed the ClassNotFoundException reported on the JIRA.

java -classpath etc/hadoop:share/hadoop/common/lib/*:share/hadoop/client/*:share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar org.apache.hadoop.examples.QuasiMonteCarlo 1 100

…tion with only shaded client jars

vinayakumarb · 2020-06-16T11:44:10Z

I ran following on hadoop dist built by mvn clean install -DskipTests -Pdist -Pnative. The patch fixed the ClassNotFoundException reported on the JIRA.
java -classpath etc/hadoop:share/hadoop/common/lib/*:share/hadoop/client/*:share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar org.apache.hadoop.examples.QuasiMonteCarlo 1 100

Thanks @iwasakims for verifying.

aajisaka · 2020-06-16T12:28:17Z

I ran the above command and verified the patch fixed the issue.
+1 pending Jenkins. Thanks.

hadoop-yetus · 2020-06-16T12:48:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	24m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 9s	Maven dependency ordering for branch
+1 💚	mvninstall	19m 44s	trunk passed
+1 💚	compile	0m 23s	trunk passed
+1 💚	mvnsite	0m 47s	trunk passed
+1 💚	shadedclient	35m 59s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 20s	the patch passed
+1 💚	compile	0m 15s	the patch passed
+1 💚	javac	0m 15s	the patch passed
+1 💚	mvnsite	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	shadedclient	13m 46s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 18s	hadoop-client-runtime in the patch passed.
+1 💚	unit	0m 18s	hadoop-client-minicluster in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		86m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/2/artifact/out/Dockerfile
GITHUB PR	#2075
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 631bed4802c3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `81d8a88`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/2/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2075/2/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

Thanx @vinayakumarb for the fix.
I verified the fix in my same Hive code and it now works with your changes :-)
+1

iwasakims

LGTM. I got no issue on running the job on minicluster with the patch.

$ HADOOP_CLASSPATH=share/hadoop/tools/lib/junit-4.12.jar bin/mapred minicluster -format -nnport 8020 -nnhttpport 9870 -rmport 8032

$ java -classpath etc/hadoop:share/hadoop/common/lib/*:share/hadoop/client/*:share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar org.apache.hadoop.examples.QuasiMonteCarlo -Dyarn.resourcemanager.address=localhost:8032 1 100

vinayakumarb · 2020-06-17T03:57:09Z

Thanks @aajisaka @iwasakims and @ayushtkn for reviews.
Merged

…tion with only shaded client jars (#2075)

…tion with only shaded client jars (apache#2075) (cherry picked from commit fc4ebb0) Conflicts: hadoop-client-modules/hadoop-client-minicluster/pom.xml hadoop-client-modules/hadoop-client-runtime/pom.xml Change-Id: I36f64efb911a16b865a6b4856bd0f8d595af1c97

…tion with only shaded client jars (apache#2075)

vinayakumarb requested review from aajisaka, ayushtkn, jojochuang and steveloughran June 15, 2020 09:20

aajisaka reviewed Jun 16, 2020

View reviewed changes

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketExcep…

66f7ddd

…tion with only shaded client jars

vinayakumarb force-pushed the bugs/YARN-10314 branch from 8c0899f to 66f7ddd Compare June 16, 2020 11:20

ayushtkn approved these changes Jun 16, 2020

View reviewed changes

iwasakims approved these changes Jun 17, 2020

View reviewed changes

vinayakumarb merged commit fc4ebb0 into apache:trunk Jun 17, 2020

asfgit pushed a commit that referenced this pull request Jun 17, 2020

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketExcep…

c1ef247

…tion with only shaded client jars (#2075)

asfgit pushed a commit that referenced this pull request Jun 17, 2020

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketExcep…

8382e31

…tion with only shaded client jars (#2075)

zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketExcep…

7eadfdc

…tion with only shaded client jars (apache#2075)

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars #2075

YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars #2075

Uh oh!

Conversation

vinayakumarb commented Jun 15, 2020

Uh oh!

hadoop-yetus commented Jun 15, 2020

Uh oh!

vinayakumarb commented Jun 15, 2020

Uh oh!

aajisaka Jun 16, 2020

Choose a reason for hiding this comment

Uh oh!

iwasakims Jun 16, 2020

Choose a reason for hiding this comment

Uh oh!

vinayakumarb Jun 16, 2020

Choose a reason for hiding this comment

Uh oh!

iwasakims commented Jun 16, 2020

Uh oh!

vinayakumarb commented Jun 16, 2020

Uh oh!

aajisaka commented Jun 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jun 16, 2020

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

iwasakims left a comment

Choose a reason for hiding this comment

Uh oh!

vinayakumarb commented Jun 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aajisaka commented Jun 16, 2020 •

edited

Loading