Closed
Description
Hi,
I've modified the example in the Quick Start Guide to use the following main function:
@file:JvmName("SimpleApp")
import org.jetbrains.spark.api.*
data class LonLat(val lon: Double, val lat: Double)
fun main() {
withSpark {
spark.dsOf(LonLat(1.0, 2.0), LonLat(3.0, 4.0)).show()
}
}
When I execute this program with:
spark-3.0.0-bin-hadoop2.7/bin/spark-submit --class "SimpleApp" --master local src/IdeaProjects/kotlin-spark-example/build/libs/kotlin-spark-example-1.0-SNAPSHOT-all.jar
I get the following error:
Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for Double
- field (class: "scala.Double", name: "lat")
- root class: "LonLat"
at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$1(KotlinReflection.scala:426)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:864)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:863)
at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:47)
at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:202)
at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$7(KotlinReflection.scala:351)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$1(KotlinReflection.scala:340)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:864)
at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:863)
at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:47)
at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:202)
at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$20(KotlinReflection.scala:470)
at org.apache.spark.sql.catalyst.DeserializerBuildHelper$.deserializerForWithNullSafetyAndUpcast(DeserializerBuildHelper.scala:54)
at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:470)
at org.apache.spark.sql.KotlinReflection.deserializerFor(KotlinReflection.scala)
at org.jetbrains.spark.api.ApiV1Kt.kotlinClassEncoder(ApiV1.kt:103)
at org.jetbrains.spark.api.ApiV1Kt.generateEncoder(ApiV1.kt:91)
at SimpleApp.main(SimpleApp.kt:47)
at SimpleApp.main(SimpleApp.kt)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
If I replace the Doubles in the data class with Ints, then the program executes correctly, printing:
+---+---+
|lat|lon|
+---+---+
| 2| 1|
| 4| 3|
+---+---+
My build environment is identical to the one at https://github.com/MKhalusova/kotlin-spark-example, and I'm using Spark 3.0.0 with Java 1.8.0_252 (AdoptOpenJDK) on MacOS 10.15.6.
Thank you,
Todd