You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-45144][BUILD] Downgrade scala-maven-plugin to 4.7.1
### What changes were proposed in this pull request?
This pr downgrade `scala-maven-plugin` to version 4.7.1 to avoid it automatically adding the `-release` option as a Scala compilation argument.
### Why are the changes needed?
The `scala-maven-plugin` versions 4.7.2 and later will try to automatically append the `-release` option as a Scala compilation argument when it is not specified by the user:
1. 4.7.2 and 4.8.0: try to add the `-release` option for Scala versions 2.13.9 and higher.
2. 4.8.1: try to append the `-release` option for Scala versions 2.12.x/2.13.x/3.1.1, and append `-java-output-version` for Scala 3.1.2.
The addition of the `-release` option has caused issues mentioned in SPARK-44376 | #41943 and #40442 (comment). This is because the `-release` option has stronger compilation restrictions than `-target`, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in the `sun.*` package are not `exports` in Java 11, 17, and 21, such as `sun.nio.ch.DirectBuffer`, `sun.util.calendar.ZoneInfo`, and `sun.nio.cs.StreamDecoder`, making them invisible when compiling across different versions.
For discussions within the Scala community, see scala/bug#12643, scala/bug#12824, scala/bug#12866, but this is not a bug.
I have also submitted an issue to the `scala-maven-plugin` community to discuss the possibility of adding additional settings to control the addition of the `-release` option: davidB/scala-maven-plugin#722.
For Apache Spark 4.0, in the short term, I suggest downgrading `scala-maven-plugin` to version 4.7.1 to avoid it automatic adding the `-release` option as a Scala compilation argument. In the long term, we should reduce use of APIs that are not `exports` for compatibility with the `-release` compilation option due to `-target` already deprecated after Scala 2.13.9.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check
run `git revert 656bf36` to revert to using Scala 2.13.11 and run `dev/change-scala-version.sh 2.13` to change Scala to 2.13
1. Run `build/mvn clean install -DskipTests -Pscala-2.13 -X` to check the Scala compilation arguments.
Before
```
[[DEBUG] [zinc] Running cached compiler 1992eaf4 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
-unchecked
-deprecation
-feature
-explaintypes
-target:jvm-1.8
-Wconf:cat=deprecation:wv,any:e
-Wunused:imports
-Wconf:cat=scaladoc:wv
-Wconf:cat=lint-multiarg-infix:wv
-Wconf:cat=other-nullary-override:wv
-Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
-Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
-Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
-Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
-Wconf:msg=method without a parameter list overrides a method with a single empty one:s
-Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
-Wconf:cat=unchecked&msg=outer reference:s
-Wconf:cat=unchecked&msg=eliminated by erasure:s
-Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
-Wconf:msg=Implicit definition should have explicit type:s
-release
8
-bootclasspath
...
```
After
```
[DEBUG] [zinc] Running cached compiler 72dd4888 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
-unchecked
-deprecation
-feature
-explaintypes
-target:jvm-1.8
-Wconf:cat=deprecation:wv,any:e
-Wunused:imports
-Wconf:cat=scaladoc:wv
-Wconf:cat=lint-multiarg-infix:wv
-Wconf:cat=other-nullary-override:wv
-Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
-Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
-Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
-Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
-Wconf:msg=method without a parameter list overrides a method with a single empty one:s
-Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
-Wconf:cat=unchecked&msg=outer reference:s
-Wconf:cat=unchecked&msg=eliminated by erasure:s
-Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
-Wconf:msg=Implicit definition should have explicit type:s
-target:8
-bootclasspath
...
```
After downgrading the version, the `-release` option should no longer appear in the compilation arguments.
2. Maven can build the project with Java 17 without the issue described in #41943. And after this pr, we can re-upgrade Scala 2.13 to Scala 2.13.11.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#42899 from LuciferYang/SPARK-45144.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
0 commit comments