ORC-1903: Support `Geometry` and `Geography` types #2032

ffacs · 2024-09-22T16:04:00Z

What changes were proposed in this pull request?

Support Geometry and Geography types for java side

Why are the changes needed?

Add support for Geometry and Geography types

How was this patch tested?

UT passed

Was this patch authored or co-authored using generative AI tooling?

YES

java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java

ffacs · 2024-09-23T13:48:11Z

cc @wgtmac @dongjoon-hyun

wgtmac · 2024-09-24T05:10:43Z

Thanks for implementing this! I will hold my review until the spec has been finalized.

dongjoon-hyun · 2024-09-25T05:52:59Z

Thank you, @ffacs .

To @wgtmac , do you mean the ORC format PR? Or, Iceberg document? Otherwise, could you provide some pointers to track this part, the spec has been finalized?

Thanks for implementing this! I will hold my review until the spec has been finalized.

dongjoon-hyun · 2024-09-25T05:55:18Z

java/pom.xml


    <mockito.version>5.10.0</mockito.version>
-    <orc-format.version>1.0.0</orc-format.version>
+    <orc-format.version>1.0.0-SNAPSHOT</orc-format.version>


? AFAIK, we didn't change anything relevant until now, do we need -SNAPSHOT?

Thank you for the review. The spec of geometry type haven't been merged into orc format yet, so I created a private package for developing here. Maybe it's a little bit early for code reviewing, let's waiting for the spec finished first.

dongjoon-hyun · 2024-09-25T05:57:03Z

java/pom.xml

    <surefire.version>3.0.0-M5</surefire.version>
    <test.tmp.dir>${project.build.directory}/testing-tmp</test.tmp.dir>
    <zstd-jni.version>1.5.6-4</zstd-jni.version>
+    <jts.version>1.19.0</jts.version>


This seems to be too old because it's released on Jun 21, 2022.

Why don't we use the latest one? Is there any incompatibility issue with something?

https://mvnrepository.com/artifact/org.locationtech.jts/jts-core/1.20.0

wgtmac · 2024-09-25T06:06:21Z

apache/parquet-format#240 is still under review and subject to change. I think we can wait until it has been finalized. @dongjoon-hyun

dongjoon-hyun · 2024-09-25T06:35:37Z

Ya, I want to check ETA for this work. Do you think we can have this for Apache ORC 2.1?

wgtmac · 2024-09-25T06:52:10Z

If 2.1.0 will be released around January 17, 2025, then I think yes.

dongjoon-hyun · 2025-01-03T16:39:40Z

Happy New Year, @ffacs and @wgtmac .

We need to release Apache ORC 2.1.0 this month.

Given that the status of Apache Parquet community, it's not finished still, right?

PARQUET-2471: Add GEOMETRY and GEOGRAPHY logical types parquet-format#240

Let me remove this from ORC 2.1.0 milestone for now, @wgtmac .

dongjoon-hyun · 2025-01-03T16:40:12Z

For the record, we can have this at Apache ORC 3.0.0.

wgtmac · 2025-01-05T13:50:00Z

Yes, the discussion of Geometry type on the Iceberg and Parquet side took longer time than expected. I don't think it will be closed soon. Moving it to 3.0.0 seems reasonable.

dongjoon-hyun · 2025-01-05T16:17:34Z

Thank you for the confirmation, @wgtmac .

dongjoon-hyun · 2025-04-10T00:02:33Z

It seems that we are ready to release Apache ORC Format v1.1.0 to support this PR.

https://github.com/apache/orc-format/milestone/2?closed=1

dongjoon-hyun · 2025-04-29T15:41:40Z

Can we resume this since the main branch is using ORC Format v1.1, @ffacs and @wgtmac ?

Apache Iceberg v1.9.0 is also released with geo type.

https://iceberg.apache.org/releases/#190-release
- Spec: Support geo type iceberg#10981

ffacs · 2025-04-30T03:32:43Z

Can we resume this since the main branch is using ORC Format v1.1, @ffacs and @wgtmac ?

Apache Iceberg v1.9.0 is also released with geo type.

https://iceberg.apache.org/releases/#190-release

Spec: Support geo type iceberg#10981

Yes, I'd start work on this these days.

dongjoon-hyun · 2025-04-30T12:59:25Z

Thank you so much!

dongjoon-hyun · 2025-06-02T18:06:26Z

Thank you for updating PR, @ffacs .

java/core/src/java/org/apache/orc/TypeDescription.java

java/core/src/test/org/apache/orc/TestOrcGeospatial.java

dongjoon-hyun · 2025-06-10T15:44:49Z

Thank you for updating, @ffacs .

java/core/src/test/org/apache/orc/TestOrcGeospatial.java

dongjoon-hyun

+1, LGTM. Thank you so much, @ffacs .

cc @williamhyun , @wgtmac , @cxzl25 , @guiyanakuang

and cc @MrPowers , @mrpowers-wb, @jiayuasu from #2224

cxzl25

LGTM

dongjoon-hyun · 2025-06-12T14:22:10Z

Thank you, @cxzl25 .

williamhyun · 2025-06-12T16:50:54Z

Thank you @ffacs, @dongjoon-hyun, @wgtmac, and @cxzl25!
Will be merging this now.

dongjoon-hyun · 2025-06-12T19:54:56Z

Thank you, @williamhyun .

ffacs · 2025-06-15T10:03:31Z

Thank you @dongjoon-hyun @williamhyun @cxzl25 .

### What changes were proposed in this pull request? This PR aims to upgrade ORC to 2.2.0. 2.2.0 RC1 is currently under voting. ### Why are the changes needed? Apache ORC 2.2.0 is a new feature release. - https://github.com/apache/orc/releases/tag/v2.2.0 - apache/orc#2032 - apache/orc#2338 - apache/orc#2269 - apache/orc#2249 - apache/orc#2144 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51676 from williamhyun/ORC-2.2.0. Authored-by: William Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added BUILD JAVA labels Sep 22, 2024

dongjoon-hyun marked this pull request as draft September 22, 2024 21:31

cxzl25 reviewed Sep 23, 2024

View reviewed changes

java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java Outdated Show resolved Hide resolved

cxzl25 reviewed Sep 23, 2024

View reviewed changes

java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Sep 25, 2024

View reviewed changes

dongjoon-hyun added this to the 2.1.0 milestone Sep 25, 2024

dongjoon-hyun removed this from the 2.1.0 milestone Jan 3, 2025

dongjoon-hyun mentioned this pull request May 15, 2025

Add support for geospatial types (geometry and geography) #2224

Closed

ffacs force-pushed the geo branch from c5fef46 to 293d918 Compare June 2, 2025 15:09

ffacs changed the title ~~[WIP] Add support for geometry type~~ ORC-1903: Add support for geospatial types Jun 2, 2025

ffacs marked this pull request as ready for review June 2, 2025 15:35

dongjoon-hyun reviewed Jun 3, 2025

View reviewed changes

java/core/src/java/org/apache/orc/TypeDescription.java Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jun 3, 2025

View reviewed changes

java/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated Show resolved Hide resolved

ffacs added 13 commits June 10, 2025 23:40

Fix spotless

d5cd485

Fix spotbugs

0338510

Fix checkstyle

4b50093

Fix spotless again

daba915

Fix some bugs

e4099f2

Fix style

866bf64

Fix building

f2c1372

Yet another fix

cedf9b6

Refactor

abb8d01

fix indentation

5bac508

revert changes in TestColumnStatisticsImpl.java

0f85db0

Refactor serdes

2219a58

inline updateBounds

79ac6ed

ffacs force-pushed the geo branch from 2cc7784 to 79ac6ed Compare June 10, 2025 15:43

dongjoon-hyun reviewed Jun 10, 2025

View reviewed changes

java/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jun 10, 2025

View reviewed changes

java/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jun 10, 2025

View reviewed changes

java/core/src/test/org/apache/orc/TestOrcGeospatial.java Outdated Show resolved Hide resolved

support Java 25-ea

95c59a5

dongjoon-hyun approved these changes Jun 11, 2025

View reviewed changes

cxzl25 approved these changes Jun 12, 2025

View reviewed changes

williamhyun approved these changes Jun 12, 2025

View reviewed changes

williamhyun closed this in 1e5c161 Jun 13, 2025

dongjoon-hyun mentioned this pull request Jul 27, 2025

[SPARK-52967][BUILD] Upgrade ORC to 2.2.0 apache/spark#51676

Closed

ORC-1903: Support Geometry and Geography types #2032

ORC-1903: Support Geometry and Geography types #2032

Uh oh!

Conversation

ffacs commented Sep 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

Uh oh!

ffacs commented Sep 23, 2024

Uh oh!

wgtmac commented Sep 24, 2024

Uh oh!

dongjoon-hyun commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

ffacs Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Sep 25, 2024

Uh oh!

dongjoon-hyun commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wgtmac commented Sep 25, 2024

Uh oh!

dongjoon-hyun commented Jan 3, 2025

Uh oh!

dongjoon-hyun commented Jan 3, 2025

Uh oh!

wgtmac commented Jan 5, 2025

Uh oh!

dongjoon-hyun commented Jan 5, 2025

Uh oh!

dongjoon-hyun commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ffacs commented Apr 30, 2025

Uh oh!

dongjoon-hyun commented Apr 30, 2025

Uh oh!

dongjoon-hyun commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

cxzl25 left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 12, 2025

Uh oh!

williamhyun commented Jun 12, 2025

Uh oh!

dongjoon-hyun commented Jun 12, 2025

Uh oh!

ffacs commented Jun 15, 2025

Uh oh!

Reviewers

Assignees

Labels

ORC-1903: Support `Geometry` and `Geography` types #2032

ORC-1903: Support `Geometry` and `Geography` types #2032

ffacs commented Sep 22, 2024 •

edited

Loading

dongjoon-hyun commented Sep 25, 2024 •

edited

Loading

dongjoon-hyun commented Sep 25, 2024 •

edited

Loading

dongjoon-hyun commented Apr 10, 2025 •

edited

Loading

dongjoon-hyun commented Apr 29, 2025 •

edited

Loading