ORC-1717: Add geometry and geography types #18

wgtmac · 2024-05-11T08:43:24Z

What changes were proposed in this pull request?

Add geometry and geography types to Apache ORC.

Why are the changes needed?

Geospatial support is a missing feature and it is supported by many popular databases, query engines, computing frameworks, etc.

How was this patch tested?

N/A

dongjoon-hyun · 2024-07-19T05:07:32Z

Thank you, @wgtmac . Is it for Apache ORC 1.1.0? If this is final, I'm +1.

dongjoon-hyun · 2024-07-19T05:08:44Z

Also, cc @williamhyun

wgtmac · 2024-07-19T05:45:56Z

@dongjoon-hyun This is not finalized yet. I will update it once ready.

dongjoon-hyun · 2024-07-19T23:26:34Z

Thank you! If it's finalized, please send a head-up email to dev@orc once more.

dongjoon-hyun · 2024-08-22T06:21:20Z

Thank you for updating. Is this ready, @wgtmac ?

wgtmac · 2024-08-22T06:23:56Z

Yes, I believe so. And I will start PoC implementations later.

dongjoon-hyun · 2024-08-22T06:25:50Z

Got it. When do you want to release orc-format v1.1?

Now to help the PoC?
After finishing PoC?

wgtmac · 2024-08-22T06:27:27Z

I think we can wait until both Java and C++ PoC implementations are finished.

dongjoon-hyun · 2024-08-22T15:31:42Z

+1 for the decision. Thank you for the confirmation.

dongjoon-hyun · 2025-04-04T18:22:08Z

Thank you. Is this ready, @wgtmac ?

wgtmac · 2025-04-07T04:00:29Z

Yes, this is ready for review @dongjoon-hyun

cc @ffacs

dongjoon-hyun · 2025-04-08T02:07:44Z

specification/ORCv2.md

 optional TimestampStatistics timestampStatistics = 9;
 optional bool hasNull = 10;
+ optional uint64 bytes_on_disk = 11;
+ optional CollectionStatistics collection_statistics = 12;


Can we handle these two above lines independently because this is irrelevant to the geometry, @wgtmac ?

I've created #22 to address this.

dongjoon-hyun · 2025-04-08T02:09:53Z

src/main/proto/orc/proto/orc_proto.proto

+// Statistics specific to Geometry or Geography type
+message GeospatialStatistics {
+  // A bounding box of geospatial instances
+  optional BoundingBox bbox = 1;


Is bbox a well-known name? I'm just wondering if we can use bounding_box like the other field names.

Yes, bbox is a well-known geospatial acronym.

dongjoon-hyun

Thank you, @wgtmac .

I have three comments.

bytes_on_disk and collection_statistics (https://github.com/apache/orc-format/pull/18/files#r2032264096)
bbox (https://github.com/apache/orc-format/pull/18/files#r2032265481)
This PR is proposed to support Apache Iceberg. However, are you sure that Apache Iceberg community keeps Apache ORC support in the next table format?

If we are not sure about (3), let's revise the PR description by avoiding mentioning Apache Iceberg community.m
I believe this PR itself has enough meaning independently to Apache ORC community. I'm +1 for this additional feature.

cc @williamhyun too

dongjoon-hyun · 2025-04-09T00:33:56Z

Gentle ping, @wgtmac .

wgtmac · 2025-04-09T01:25:37Z

Are you sure that Apache Iceberg community keeps Apache ORC support in the next table format?

I'm not sure about the long-term goal. The recent discussion w.r.t the file format API still considers Apache ORC: https://lists.apache.org/thread/ovyh52m2b6c1hrg4fhw3rx92bzr793n2

dongjoon-hyun · 2025-04-09T03:03:54Z

Thank you, @wgtmac .

I'm referring the latest Apache Iceberg community sync-up meeting on March 16th which superseded the above discussion on February 12th.

https://youtu.be/9BBZKTfcU0s?t=44m10s

wgtmac · 2025-04-09T04:04:27Z

Thanks for the info! I just watched that video and understand the concern from the Iceberg community. There are several issues like missing features (variant, geometry, etc.) and implementation (default values, schema evolution, etc.) For the latter issue, does anyone in the Apache ORC community know the detail?

dongjoon-hyun

+1, LGTM. Thank you for updating, @wgtmac .

dongjoon-hyun · 2025-04-09T23:51:52Z

Since this is an improvement without a breaking change, shall we release as Apache ORC v1.1.0? WDYT, @wgtmac ?

wgtmac · 2025-04-10T01:38:33Z

@dongjoon-hyun Sounds good! I think this can help the implementation a lot.

### What changes were proposed in this pull request? This PR aims to upgrade ORC Format to 1.1.0. ### Why are the changes needed? To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #2190 from dongjoon-hyun/ORC-1876. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to upgrade ORC Format to 1.1.0. ### Why are the changes needed? To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #2190 from dongjoon-hyun/ORC-1876. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit f5e2413) Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to upgrade Apache ORC Format to 1.1.0 from 1.1.0 for Apache Spark 4.1.0. ### Why are the changes needed? Apache ORC Format v1.1.0 is released on 2025-04-18. - https://github.com/apache/orc-format/releases/tag/v1.1.0 To bring the latest feature and bug fixes. - https://github.com/apache/orc-format/milestone/2?closed=1 - apache/orc-format#18 - apache/orc-format#19 ### Does this PR introduce _any_ user-facing change? No, there is no behavior change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50588 from dongjoon-hyun/SPARK-51801. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added DOCS PROTO labels May 11, 2024

wgtmac marked this pull request as ready for review May 12, 2024 08:15

wgtmac changed the title ~~WIP: Add geometry type~~ ORC-1717: Add geometry type May 12, 2024

cholmes mentioned this pull request May 24, 2024

Thoughts about a first-class GEOMETRY data type in Parquet? opengeospatial/geoparquet#222

Closed

dongjoon-hyun mentioned this pull request Sep 25, 2024

ORC-1903: Support Geometry and Geography types apache/orc#2032

Closed

wgtmac changed the title ~~ORC-1717: Add geometry type~~ ORC-1717: Add geometry and geography types Apr 3, 2025

wgtmac force-pushed the geo branch from d03fe4c to e3d3957 Compare April 3, 2025 06:02

dongjoon-hyun reviewed Apr 8, 2025

View reviewed changes

wgtmac added 3 commits April 9, 2025 11:53

WIP: Add geometry type

e25acb9

reflect latest change from parquet proposal

05a09ac

reflect the latest parquet spec

753bffc

wgtmac force-pushed the geo branch from e3d3957 to 753bffc Compare April 9, 2025 03:54

dongjoon-hyun approved these changes Apr 9, 2025

View reviewed changes

dongjoon-hyun merged commit e807a18 into apache:main Apr 9, 2025
2 checks passed

dongjoon-hyun added this to the 1.1.0 milestone Apr 9, 2025

This was referenced Apr 15, 2025

[SPARK-51801][BUILD] Upgrade ORC Format to 1.1.0 apache/spark#50588

Closed

ORC-1876: Upgrade to ORC Format 1.1 apache/orc#2190

Closed

dongjoon-hyun mentioned this pull request May 15, 2025

Add support for geospatial types (geometry and geography) apache/orc#2224

Closed

ORC-1717: Add geometry and geography types #18

ORC-1717: Add geometry and geography types #18

Uh oh!

Conversation

wgtmac commented May 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

wgtmac commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Jul 19, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

wgtmac commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

wgtmac commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Aug 22, 2024

Uh oh!

dongjoon-hyun commented Apr 4, 2025

Uh oh!

wgtmac commented Apr 7, 2025

Uh oh!

dongjoon-hyun Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

wgtmac Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

wgtmac Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 9, 2025

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 9, 2025

Uh oh!

wgtmac commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wgtmac commented May 11, 2024 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

wgtmac commented Apr 9, 2025 •

edited

Loading