Upgrade Scala version from 2.13.8 to 2.13.14

[dupen01]

🔍 Description

Issue References 🔗

To support Java 21, the Scala version needs to be upgraded to 2.13.11 or above.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[pan3793]

can we use Scala 2.13.14 to align with Spark 4.0.0-preview1?

[pan3793]

BTW, please rebase your code on latest master branch, there were some issues on CI in my previous commits

[pan3793]

keep our dependencies up to date is a regular work, we don't need this comment

[pan3793]

the failure should be fixed by upgrade maven.plugin.silencer.version from 1.7.13 to 1.7.17

[dupen01]

the failure should be fixed by upgrade maven.plugin.silencer.version from 1.7.13 to 1.7.17

yes, I will upgrade maven.plugin.silencer.version to 1.7.17

[pan3793]

LGTM if CI pass, thanks for your contribution

[pan3793]

the failure is not a real issue, let's follow apache/spark#46288 to add ignore classes

[pan3793]

Seems FinalStageResourceManagerSuite gets stuck after upgrading Scala 2.13, cc @zhouyifan279 do you have time to take a look? thanks in advance

[bowenliang123]

Why we have to update the silencer version?

[pan3793]

each version of silencer has a strictly supported Scala version list, it's common to update with Scala version together
https://mvnrepository.com/artifact/com.github.ghik/silencer-plugin

[bowenliang123]

Cool. SGTM. Thanks for the explanation.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (df8448c) to head (541f860).
Report is 2 commits behind head on master.

Files	Patch %	Lines
...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6603   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         680     680           
  Lines       42045   42046    +1     
  Branches     5739    5739           
======================================
- Misses      42045   42046    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

I checked the failure message from FinalStageResourceManagerSuite

16:21:05.237 ERROR org.apache.spark.network.server.TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 9220236832923458123
java.io.InvalidClassException: scala.concurrent.duration.FiniteDuration; local class incompatible: stream classdesc serialVersionUID = -4594686286536372853, local class serialVersionUID = -6513803676778706429
        at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:597)
        ...
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:87)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:123)
        at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$deserialize$2(NettyRpcEnv.scala:299)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
        at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:352)

the root cause is that it launches a local cluster using SPARK_HOME, thus using a different Scala library with SPARK_HOME may cause serialization issues as JavaSerializer is not used here.

[pan3793]

Thanks, merged to master