Plan for Tuples/Arity's?

[Jolanrensen]

This is more of a future proofing question than an actual issue, but I was wondering if there was any plan regarding the use of Tuples in the library.
On the one hand, many of Sparks functions return Scala Tuples, which work fine although could use with some operator funs for destructuring and operator overloads.
On the other hand, the library provides the Arity's (odd name but sure) data classes which have the operator plus funs but there is no conversion between those and the Scala Tuples (yet, but I could create a pull request easily).
And finally there are the Kotlin Pairs and Tuples...

So what's the future plan? Personally I'd be fine with providing operator funs for the Scala Tuples and make them a bit more Kotlin friendly (I've got componentX operator funs ready to pull request if needed) and losing the Arity's. (Maybe make the c() functions create Scala Tuples too)
But currently, as user of the library, I'm constantly trying to figure out which method to use for Tuples.

[Jolanrensen]

I created a lot of extension functions for Tuples in Kotlin and created a separate repo for it here. It provides tuple destructuring, textual accessing of values, easy extending of tuples using '..' and tuple joining using '+', as well as some quality of life extensions like iterating over them etc.
I think it would be a very welcome extension to the library, but it might also be useful as a library on its own.
Anyways, you can take a look and please let me know what you think and if you would change anything.

[asm0dey]

Sorry for not answering for a long time, it looks like I somehow don't receive notification from github O_O
They are already implemented here: https://github.com/JetBrains/kotlin-spark-api/blob/master/kotlin-spark-api/common/src/main/kotlin/org/jetbrains/kotlinx/spark/api/VarArities.kt

You can create unnamed tuple with shortcut function c and you can even add them one to another (to certain extent of course)

[Jolanrensen]

Sorry for not answering for a long time, it looks like I somehow don't receive notification from github O_O
They are already implemented here: https://github.com/JetBrains/kotlin-spark-api/blob/master/kotlin-spark-api/common/src/main/kotlin/org/jetbrains/kotlinx/spark/api/VarArities.kt

You can create unnamed tuple with shortcut function c and you can even add them one to another (to certain extent of course)

@asm0dey
Like I said here #79
the Spark API can still return Tuples (like joinWith()), which the Kotlin Spark API can then not encode.
Also, your project might contain Java or Scala parts in which Tuples are more common, so not being able to provide an encoder for them would make interoperability worse.

[Jolanrensen]

@asm0dey (just making sure you still get the notification, I'm not trying to be pushy, I promise :P)

[asm0dey]

Getting it!

[asm0dey]

@Jolanrensen question about joinWith: currently we have wrappers for all the types of joins: inner, left, right, which are supported by joinWith. It's supposed to call those methods. Do you think it's not enough?

[Jolanrensen]

@asm0dey Oh right, I see, I think for that specific example it's alright then...
I am however wondering why you chose to convert the value to a Pair instead of an Arity, as that then attempts to be the replacement of Scala's Tuples (this also applies to reduceGroups). Plus, doesn't converting back and forth from Arity/Pair to Scala Tuples in the API create unnecessary / unexpected overhead?

Another function which still returns Scala Tuples is Dataset.select() btw, but of course, these can be added to the Kotlin Spark API with converters.

However, my point about interoperability with Java and Scala code in the same project still holds. If a function from somewhere in the project returns a Tuple, then the Kotlin Spark API cannot use them in any of the function that require the implicit encoder<>().

[asm0dey]

Well, I've selected Pair/Triple because they're idiomatic for Kotlin. And I hope that single conversion from Tuple2 to Pair costs almost nothing because of escape analysis. Do I have a place where I'm creating Scala tuple from Pair?

But yes, select creates tuples, it's true and it means you're right and we should support them in both Spark 2 and 3.

How do you think, what is the best way to implement support for tuples?

[Jolanrensen]

@asm0dey
I see... However, Spark functions are often required to be as fast as possible so they can be scaled up. If the cost of conversion to Pairs is negligible then it's okay, but this might need some speed testing to make sure :).

I don't think you're converting to Scala Tuples at the moment, so that's good (so not back "and forth", haha).

My idea was to at least support Tuple encoding, which I made working here: #79
That way Tuples work in all functions that require the encoder<>() function. We might also require a Tuple DataType or schema, but that's where I got stuck.

I see where you're coming from choosing Pairs and Triples, because they're native to Kotlin, however this does mean the user can now use Pairs, Triples, Arity's and Tuples from Scala... So that might be overwhelming in terms of options. I'm currently using my own library (https://github.com/Jolanrensen/ScalaTuplesInKotlin) which tries to make Scala's Tuples more manageable from Kotlin. That way I don't have to decide whether to use Pairs, Triples etc. Not sure if that's the way to go, but yeah. What do you think?

[asm0dey]

@Jolanrensen

It looks like we should call Scala reflection for Product classes and obtain schema there. For example this method https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L712 should work with Tuples perfectly.

Would you like to try this approach?

[Jolanrensen]

@asm0dey I can give it a try perhaps!
I see this file is also mirrored as KotlinReflection.scala in the Scala part of the API. Can we also use this?

[Jolanrensen]

@asm0dey
Tuple encoding seems to work using my pull request similar to how data classes are encoded. You can give it a look if you like :).

But what do you think about the pair/triple/arity/tuple situation?