Skip to content

Conversation

@Kontinuation
Copy link
Member

@Kontinuation Kontinuation commented Dec 17, 2025

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

This PR refactors several spatial aggregate functions to fix NULL handling. Impacted functions are

  • ST_Envelope_Aggr: fixed null handling, simplified the implementation by switching to lightweight intermediate data structure
  • ST_Union_Aggr: fixed null handling
  • ST_Intersection_Aggr: fixed null handling
  • ST_Collect_Agg: return null instead of empty geometry collection when all the inputs are null. This is theoretically a breaking change, but it is a niche case so hopefully it won't impact many users.

How was this patch tested?

Added new tests and tested locally using a standalone cluster to make sure that we are not damaging the serde in distributed environment.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@Kontinuation Kontinuation changed the title [WIP] Fix NULL handling for various aggregation functions in SedonaSpark [GH-2565] Fix NULL handling for various aggregation functions in SedonaSpark Dec 17, 2025
@Kontinuation Kontinuation force-pushed the fix-aggr-null-handling branch from fbf1027 to cfb6c9d Compare December 17, 2025 10:19
@Kontinuation Kontinuation marked this pull request as ready for review December 17, 2025 10:20
@Kontinuation
Copy link
Member Author

The Python build extension failure is probably caused by the newly released PySpark 4.1.0 not playing well with Windows. See apache/spark#50466

@jiayuasu jiayuasu merged commit fa32f44 into apache:master Dec 17, 2025
36 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Aggregation functions in SedonaSpark throw exception when processing NULL values

2 participants