Pass Field information back and forth when using scalar UDFs

[timsaucer]

Which issue does this PR close?

Closes #1172

Rationale for this change

Since we now have the ability to pass Field information instead of just DataType with ScalarUDFs, this feature adds similar support for Python written UDFs. Without this feature you must write your UDFs in rust and expose them to Python. This enhancement greatly expands the use cases where PyArrow data can be leveraged.

What changes are included in this PR?

• Adds a ScalarUDF implementation for python based UDFs instead of relying on the create_udf function
• Adds support for converting to pyarrow array via FFI but including the Field schema instead of the data type

Are there any user-facing changes?

This expands on the current API and is backwards compatible.

TODO

• Add unit tests
• Update online documentation
• Update python signatures (type hints)

[timsaucer]

@kosiew Here is an alternate approach. Instead of relying on extension type features it is going to pass the Field information when creating the FFI array. This will capture pyarrow extensions as well as any other metadata that any user assigns on the input.

I'm going to leave it in draft until I can finish up those additional items on my check list.

What do you think?

cc @paleolimbot

[paleolimbot]

What do you think?

Definitely! Passing the argument fields/return fields should do it. Using __arrow_c_schema__ might be more flexible than isinstance(x, pa.Field) (arro3, nanoarrow, and polars types would work too).

We have a slightly different signature model in SedonaDB ("type matchers") because the existing signature matching doesn't consider metadata, but at the Arrow/FFI level we're doing approximately the same thing: apache/sedona-db#228 . We do use the concept of SedonaType for arguments and return type (but these are serializable to/deserializable from fields).

[kosiew]

is the use of PyArrow's private _import_from_c advisable?

[timsaucer]

This code is a near duplicate of how we already convert ArrayData into a pyarrow object. You can see the original here. The difference in this function is that we know the field instead of only the data type.

[paleolimbot]

A more modern way is to use __arrow_c_schema__ (although I think import_from_c will be around for a while). It's only a few lines:

https://github.com/apache/sedona-db/blob/main/python/sedonadb/src/import_from.rs#L151-L157

[timsaucer]

Also worth evaluating while we're doing this: For scalar values, is it possible for them to contain metadata? If I do pa.scalar(uuid.uuid4().bytes, type=pa.uuid()) and I check the type I should have the extension data. Maybe this is already supported, but as part of this PR I want to evaluate that as well.

Opened new issue so there isn't too much scope creep