[SPARK-34007][BUILD] Downgrade scala-maven-plugin to 4.3.0 #31031
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@maropu, @cloud-fan @dongjoon-hyun, @srowen is anyone online? hope we can merge this quick and I restart RC process. |
cloud-fan
approved these changes
Jan 5, 2021
Member
|
LGTM |
Member
Author
|
I manually verified this change. I will merge and keep monitoring the build. Merged to master and branch-3.1. |
HyukjinKwon
added a commit
that referenced
this pull request
Jan 5, 2021
### What changes were proposed in this pull request? This PR is a partial revert of #30456 by downgrading scala-maven-plugin from 4.4.0 to 4.3.0. Currently, when you run the docker release script (`./dev/create-release/do-release-docker.sh`), it fails to compile as below during incremental compilation with zinc for an unknown reason: ``` [INFO] Compiling 21 Scala sources and 3 Java sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes ... [ERROR] ## Exception when compiling 24 sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes java.lang.SecurityException: class "javax.servlet.SessionCookieConfig"'s signer information does not match signer information of other classes in the same package java.lang.ClassLoader.checkCerts(ClassLoader.java:891) java.lang.ClassLoader.preDefineClass(ClassLoader.java:661) java.lang.ClassLoader.defineClass(ClassLoader.java:754) java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) java.net.URLClassLoader.defineClass(URLClassLoader.java:468) java.net.URLClassLoader.access$100(URLClassLoader.java:74) java.net.URLClassLoader$1.run(URLClassLoader.java:369) java.net.URLClassLoader$1.run(URLClassLoader.java:363) java.security.AccessController.doPrivileged(Native Method) java.net.URLClassLoader.findClass(URLClassLoader.java:362) java.lang.ClassLoader.loadClass(ClassLoader.java:418) java.lang.ClassLoader.loadClass(ClassLoader.java:351) java.lang.Class.getDeclaredMethods0(Native Method) java.lang.Class.privateGetDeclaredMethods(Class.java:2701) java.lang.Class.privateGetPublicMethods(Class.java:2902) java.lang.Class.getMethods(Class.java:1615) sbt.internal.inc.ClassToAPI$.toDefinitions0(ClassToAPI.scala:170) sbt.internal.inc.ClassToAPI$.$anonfun$toDefinitions$1(ClassToAPI.scala:123) scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86) sbt.internal.inc.ClassToAPI$.toDefinitions(ClassToAPI.scala:123) sbt.internal.inc.ClassToAPI$.$anonfun$process$1(ClassToAPI.scala:3 ``` This happens when it builds Spark with Hadoop 2. It doesn't reproduce when you build this alone. It should follow the sequence of build in the release script. This is fixed by downgrading. Looks like there is a regression in scala-maven-plugin somewhere between 4.4.0 and 4.3.0. ### Why are the changes needed? To unblock the release. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? It can be tested as below: ```bash ./dev/create-release/do-release-docker.sh -d $WORKING_DIR ``` Closes #31031 from HyukjinKwon/SPARK-34007. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 356fdc9) Signed-off-by: HyukjinKwon <[email protected]>
|
Test build #133661 has finished for PR 31031 at commit
|
Member
|
Oh, thanks! |
3 tasks
yaooqinn
pushed a commit
to apache/kyuubi
that referenced
this pull request
Mar 16, 2021
 [](https://github.com/yaooqinn/kyuubi/pull/441)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> There are several known issues with `scala-maven-plugin:4.4.0` about `zinc`. On CentOS 7, with Oracle JDK 8 installed via RPM, `/bin/java` links to `jdk/jre/bin/java` but `/bin/javac` links to `jdk/bin/javac` ``` [opsbd-ck-uat-30007-vm ~]$ ll -l /bin/java lrwxrwxrwx 1 root root 22 Mar 12 14:03 /bin/java -> /etc/alternatives/java [opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/java lrwxrwxrwx 1 root root 41 Mar 12 14:03 /etc/alternatives/java -> /usr/java/jdk1.8.0_251-amd64/jre/bin/java [opsbd-ck-uat-30007-vm ~]$ ll -l /bin/javac lrwxrwxrwx 1 root root 23 Mar 12 14:03 /bin/javac -> /etc/alternatives/javac [opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/javac lrwxrwxrwx 1 root root 38 Mar 12 14:03 /etc/alternatives/javac -> /usr/java/jdk1.8.0_251-amd64/bin/javac ``` when run `build/dist --tgz` cause ``` [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile (scala-test-compile-first) on project kyuubi-common: wrap: java.io.IOException: Cannot run program "/usr/java/jdk1.8.0_221-amd64/jre/bin/javac" (in directory "/data/ansible/ci/kyuubi"): error=2, No such file or directory -> [Help 1] ``` And there is another issue reported by Apache Spark: apache/spark#31031 Please also merge it into branch-1.1 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #441 from pan3793/build. 6f6c34d [Cheng Pan] add comments 312b2c5 [Cheng Pan] [BUILD] Downgrade scala-maven-plugin to 4.3.0 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Kent Yao <[email protected]>
yaooqinn
pushed a commit
to apache/kyuubi
that referenced
this pull request
Mar 16, 2021
 [](https://github.com/yaooqinn/kyuubi/pull/441)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> There are several known issues with `scala-maven-plugin:4.4.0` about `zinc`. On CentOS 7, with Oracle JDK 8 installed via RPM, `/bin/java` links to `jdk/jre/bin/java` but `/bin/javac` links to `jdk/bin/javac` ``` [opsbd-ck-uat-30007-vm ~]$ ll -l /bin/java lrwxrwxrwx 1 root root 22 Mar 12 14:03 /bin/java -> /etc/alternatives/java [opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/java lrwxrwxrwx 1 root root 41 Mar 12 14:03 /etc/alternatives/java -> /usr/java/jdk1.8.0_251-amd64/jre/bin/java [opsbd-ck-uat-30007-vm ~]$ ll -l /bin/javac lrwxrwxrwx 1 root root 23 Mar 12 14:03 /bin/javac -> /etc/alternatives/javac [opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/javac lrwxrwxrwx 1 root root 38 Mar 12 14:03 /etc/alternatives/javac -> /usr/java/jdk1.8.0_251-amd64/bin/javac ``` when run `build/dist --tgz` cause ``` [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile (scala-test-compile-first) on project kyuubi-common: wrap: java.io.IOException: Cannot run program "/usr/java/jdk1.8.0_221-amd64/jre/bin/javac" (in directory "/data/ansible/ci/kyuubi"): error=2, No such file or directory -> [Help 1] ``` And there is another issue reported by Apache Spark: apache/spark#31031 Please also merge it into branch-1.1 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #441 from pan3793/build. 6f6c34d [Cheng Pan] add comments 312b2c5 [Cheng Pan] [BUILD] Downgrade scala-maven-plugin to 4.3.0 Authored-by: Cheng Pan <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 794a753) Signed-off-by: Kent Yao <[email protected]>
<!--](https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!--){kind=link}
gengliangwang
added a commit
that referenced
this pull request
Aug 20, 2021
### What changes were proposed in this pull request? When preparing Spark 3.2.0 RC1, I hit the same issue of #31031. ``` [INFO] Compiling 21 Scala sources and 3 Java sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes ... [ERROR] ## Exception when compiling 24 sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes java.lang.SecurityException: class "javax.servlet.SessionCookieConfig"'s signer information does not match signer information of other classes in the same package java.lang.ClassLoader.checkCerts(ClassLoader.java:891) java.lang.ClassLoader.preDefineClass(ClassLoader.java:661) ``` This PR is to apply the same fix again by downgrading scala-maven-plugin to 4.3.0 ### Why are the changes needed? To unblock the release process. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Build test Closes #33791 from gengliangwang/downgrade. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Gengliang Wang <[email protected]>
gengliangwang
added a commit
that referenced
this pull request
Aug 20, 2021
### What changes were proposed in this pull request? When preparing Spark 3.2.0 RC1, I hit the same issue of #31031. ``` [INFO] Compiling 21 Scala sources and 3 Java sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes ... [ERROR] ## Exception when compiling 24 sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes java.lang.SecurityException: class "javax.servlet.SessionCookieConfig"'s signer information does not match signer information of other classes in the same package java.lang.ClassLoader.checkCerts(ClassLoader.java:891) java.lang.ClassLoader.preDefineClass(ClassLoader.java:661) ``` This PR is to apply the same fix again by downgrading scala-maven-plugin to 4.3.0 ### Why are the changes needed? To unblock the release process. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Build test Closes #33791 from gengliangwang/downgrade. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Gengliang Wang <[email protected]> (cherry picked from commit f0775d2) Signed-off-by: Gengliang Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR is a partial revert of #30456 by downgrading scala-maven-plugin from 4.4.0 to 4.3.0.
Currently, when you run the docker release script (
./dev/create-release/do-release-docker.sh), it fails to compile as below during incremental compilation with zinc for an unknown reason:This happens when it builds Spark with Hadoop 2. It doesn't reproduce when you build this alone. It should follow the sequence of build in the release script.
This is fixed by downgrading. Looks like there is a regression in scala-maven-plugin somewhere between 4.4.0 and 4.3.0.
Why are the changes needed?
To unblock the release.
Does this PR introduce any user-facing change?
No, dev-only.
How was this patch tested?
It can be tested as below:
./dev/create-release/do-release-docker.sh -d $WORKING_DIR