LightGBMRanker in example notebook throws `java.io.InvalidClassException: org.apache.spark.ml.Ranker`

[anishazaveri]

Describe the bug
I'm running the LightGBMRanker code (Cmds 37-41) in example notebook LightGBM - Overview.ipynb from here on Azure Databricks. I get a bug in command lgbm_ranker_model = lgbm_ranker.fit(df) (Cmd 41)

Info (please complete the following information):

• SynapseML Version: 2.12
• Spark Version 3.1.2
• Spark Platform Databricks DBR 9.1 LTS
• Installation co-ordinations: com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT

Stacktrace

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 654.0 failed 4 times, most recent failure: Lost task 0.3 in stage 654.0 (TID 217219) (10.246.66.12 executor 72): java.io.InvalidClassException: org.apache.spark.ml.Ranker; local class incompatible: stream classdesc serialVersionUID = -3133757800595162776, local class serialVersionUID = 8149322387161021528

[imatiach-msft]

@anishazaveri
Interesting, I haven't seen this error before, we may need to add this:
https://github.com/microsoft/SynapseML/blob/master/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/booster/LightGBMBooster.scala#L210

Perhaps it needs to be added here:
https://github.com/microsoft/SynapseML/blob/master/core/src/main/scala/org/apache/spark/ml/Ranker.scala#L17

I'm not quite sure, as I haven't seen this error in the tests that run on a cluster for some reason.

[imatiach-msft]

@anishazaveri what is your spark setup? Based on this comment:
apache/spark#629 (comment)
it seems like it may be due to having two spark versions loaded somehow. Or perhaps there are two synapseml versions loaded. Regardless, perhaps we should add the SerialVersionUID just in case for serialized classes.

[imatiach-msft]

Based on the fact you are running on AzureDatabricks I'm not really sure that is the issue. Wait, actually, can you show what libraries you have installed? I wonder if somehow you have two synapseml libraries installed in the databricks cluster at the same time? Perhaps ensuring you only have one version installed and restarting the cluster would fix this?

[anishazaveri]

I do have an older mmlspark version installed as well on the cluster. I will uninstall it and report back

[imatiach-msft]

@anishazaveri did you have a chance to uninstall, to see if the issue is fixed?

[anishazaveri]

Yes, the uninstallation fixed the issue. Thank you!