Proof-of-concept Parquet GEOMETRY logical type implementation #43977

Kontinuation · 2024-09-06T02:23:14Z

Rationale for this change

This is a continuation of #43196

In apache/parquet-format#240 a GEOMETRY logical type for Parquet is proposed with a proof-of-concept Java implementation ( apache/parquet-java#1379 ). This is a PR to explore what an implementation would look like in C++.

What changes are included in this PR?

We are still in progress of completing all necessary changes to integrate geometry logical type support to the C++ implementation.

[DONE] Adding geometry logical type
[DONE] Adding geometry column statistics
[DONE] Support reading/writing parquet files containing geometry columns

Are these changes tested?

The tests added only cover very basic use cases. Comprehensive tests will be added in future commits.

Are there any user-facing changes?

Yes! (And will eventually be documented)

Co-authored-by: Gang Wu <[email protected]>

Kontinuation · 2024-09-17T00:07:57Z

Please ping me when ready for review again. Thanks!

@wgtmac I have added ColumnIndex and covering support. Please help review this PR when you have time, thank you.

wgtmac

I just finished reviewing it for the 2nd pass. Thanks for the great work!

My main concern is the difference with Java PoC, which generates min/max values in the statistics and page index as if the GEOMETRY column is a pure BYTE_ARRAY column. Otherwise we need to revise the spec to add a lot of exceptions for geometry type. WDYT?

cpp/src/parquet/thrift_internal.h

cpp/src/parquet/column_reader.cc

cpp/src/parquet/CMakeLists.txt

wgtmac · 2024-09-14T01:32:43Z

cpp/src/parquet/types.cc

+  Geometry(std::string crs, LogicalType::GeometryEdges::edges edges,
+           LogicalType::GeometryEncoding::geometry_encoding encoding,
+           std::string metadata)
+      : LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),


Suggested change

: LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),

: LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNSIGNED),

SortOrder::UNSIGNED is the default sort order of BYTE_ARRAY type. Could we just use this so you don't have to change a line in column_writer.cc. The good thing is that ColumnIndex of geometry type can also be generated automatically, though the min/max values are derived from their binary values and useless. This is the same practice used in the Java PoC impl.

This is life-saving. Now all the special handling of geometry statistics for unknown sort order has gone away.

cpp/src/parquet/types.cc

cpp/src/parquet/reader_test.cc

cpp/src/parquet/statistics.cc

wgtmac · 2024-09-18T15:02:31Z

cpp/src/parquet/statistics.cc

+    out.mmax = maxes[3];
+
+    if (coverings_.empty()) {
+      // Generate coverings from bounding box if coverings is not present


When will coverings_ be empty? Is it the default behavior? I'm not sure if we need to check if the edges is planar since bbox is not accurate for spherical edges. BTW, if we don't have a good implementation for coverings, I think we can just ignore it for now.

This is for generating coverings from the bounding box when assembling the encoded representation of the geometry statistics. I've added a member called generate_covering_ to make it more explicit.

wgtmac · 2024-09-18T15:03:44Z

cpp/src/parquet/statistics.h

+
+class GeometryStatisticsImpl;
+
+class PARQUET_EXPORT GeometryStatistics {


Thanks for adding theses!

wgtmac · 2024-09-18T15:07:07Z

cpp/src/parquet/statistics.h

-  return std::static_pointer_cast<TypedStatistics<DType>>(Statistics::Make(
-      descr, encoded_min, encoded_max, num_values, null_count, distinct_count,
-      has_min_max, has_null_count, has_distinct_count, pool));
+    int64_t distinct_count, const EncodedGeometryStatistics& geometry_statistics,


Why not directly add const EncodedGeometryStatistics* geometry_statistics = NULLPTR to the end of the existing function signature?

…several review comments

… coverings from bounding box when populating the encoded statistics

…and upper-right points

Kontinuation · 2024-09-19T13:04:12Z

I just finished reviewing it for the 2nd pass. Thanks for the great work!

My main concern is the difference with Java PoC, which generates min/max values in the statistics and page index as if the GEOMETRY column is a pure BYTE_ARRAY column. Otherwise we need to revise the spec to add a lot of exceptions for geometry type. WDYT?

I've changed the min/max statistics of geometry columns to be the WKB representation of lower-left and upper-right corners in the last commit according to apache/iceberg#10981 (comment).

wgtmac

Thanks for the quick change! I've left some minor comments.

wgtmac · 2024-09-19T15:48:07Z

cpp/src/parquet/column_reader.cc

+    EncodedGeometryStatistics encoded_geometry_stats;
+    if (stats.__isset.geometry_stats) {
+      encoded_geometry_stats = FromThrift(stats.geometry_stats);
+    }
+    page_statistics.set_geometry(encoded_geometry_stats);


Suggested change

EncodedGeometryStatistics encoded_geometry_stats;

if (stats.__isset.geometry_stats) {

encoded_geometry_stats = FromThrift(stats.geometry_stats);

}

page_statistics.set_geometry(encoded_geometry_stats);

page_statistics.set_geometry(FromThrift(stats.geometry_stats));

wgtmac · 2024-09-19T16:08:37Z

cpp/src/parquet/statistics.h

-  return std::static_pointer_cast<TypedStatistics<DType>>(Statistics::Make(
-      descr, encoded_min, encoded_max, num_values, null_count, distinct_count,
-      has_min_max, has_null_count, has_distinct_count, pool));
+    int64_t distinct_count, const EncodedGeometryStatistics& geometry_statistics,


IMO, we'd better not to add another overload. If there is a compelling reason to do so, we can add a ARROW_DEPRECATED macro to the old one.

1. Remove metadata property of geometry logical types 2. Remove covering from geometry statistics

1. geometry statistics moved out of statistics, it is now a field of column metadata 2. geometry statistics is removed from page index

emkornfield · 2025-02-06T19:49:35Z

cpp/src/parquet/geometry_util_internal.h

+    }
+
+    uint32_t value;
+    memcpy(&value, data_, sizeof(uint32_t));


use SafeLoadAs

emkornfield · 2025-02-06T19:59:32Z

cpp/src/parquet/geometry_util_internal.h

+  WKBGenericSequenceBounder(const WKBGenericSequenceBounder&) = default;
+
+  void ReadPoint(WKBBuffer* src, Dimensions::dimensions dimensions, bool swap) {
+    if (ARROW_PREDICT_TRUE(!swap)) {


are there benchmarks that show the swap template variable adds a lot of perfomance. I'd expect branch prediction to do a pretty good job here?

reading the comment above, I see that might be an open question.

I refactored the utility file down to about 400 lines in #45459 (and removed all the templating!)

emkornfield · 2025-02-06T20:02:33Z

cpp/src/parquet/geometry_util_internal.h

+  ptr[0] = kWkbNativeEndianness;
+  uint32_t geom_type = geometry::GeometryType::ToWKB(
+      geometry::GeometryType::geometry_type::POINT, has_z, has_m);
+  memcpy(&ptr[1], &geom_type, 4);


SafeLoadAs?

Done in #45459 !

emkornfield · 2025-02-06T20:03:15Z

cpp/src/parquet/geometry_util_internal.h

+
+namespace parquet::geometry {
+
+constexpr double kInf = std::numeric_limits<double>::infinity();


small comment here, for new code, it might be nicer to use Arrow Status, so that this can potentially be moved to the main arrow library for things not directly related to parquet (at least WKB generating/parsing parts)

Fixed in #45459 !

…implementations (#45459) ### Rationale for this change The GEOMETRY and GEOGRAPHY logical types are being proposed as an addition to the Parquet format. ### What changes are included in this PR? This is a continuation of @ Kontinuation 's initial PR (#43977) implementing apache/parquet-format#240 , which included: - Added geometry logical types (printing, serialization, deserialization) - Added geometry column statistics (serialization, deserialization, writing) - Support reading/writing parquet files containing geometry columns Changes after this were: - Rebasing on the latest apache/arrow - Split geography/geometry types - Synchronize the final parameter names (e.g., no more "encoding", "edges" -> "algorithm") - Simplify geometry_util_internal.h and use Status instead of exceptions according to suggestions from the previous PR In order to write test files, I also: - Implemented conversion to/from the GeoArrow extension type - Wired the requisite options to pyarrow so that the files could be written from Python Those last two are probably a bit much for this particular PR, and I'm happy to move them. Some things that aren't in this PR (but should be in this one or a future PR): - Update the bounding box logic to implement the "wraparound" bounding boxes where `max > min` (and generally make sure the stats for geography are written for trivial cases) - Test more invalid WKB cases ### Are these changes tested? Yes! ### Are there any user-facing changes? Yes! Example from the included Python bindings: ```python import geopandas import geopandas.testing import geoarrow.pyarrow as _ # for extension type registration import pyarrow as pa from pyarrow import parquet # More example files at # https://github.com/geoarrow/geoarrow-data gdf = geopandas.read_file( "https://raw.githubusercontent.com/geoarrow/geoarrow-data/v0.2.0-rc6/example-crs/files/example-crs_vermont-utm.fgb" ) gdf.total_bounds #> array([ 625858.19400305, 4733644.25036889, 775539.58040423, #> 4989817.92403143]) gdf.crs.to_authority() #> ('EPSG', '32618') tab = pa.table(gdf.to_arrow()) # Use store_schema=False to explicitly check conversion to Parquet LogicalType # This example also works with store_schema=True (the default) and without # an explicit arrow_extensions_enabled=True on read. parquet.write_table(tab, "vermont.parquet", store_schema=False) f = parquet.ParquetFile("vermont.parquet", arrow_extensions_enabled=True) f.schema #> <pyarrow._parquet.ParquetSchema object at 0x1402e5940> #> required group field_id=-1 schema { #> optional binary field_id=-1 geometry (Geometry(crs={"type":"ProjectedCRS", ...})); #> } f.metadata.row_group(0).column(0).geo_statistics #> <pyarrow._parquet.GeoStatistics object at 0x127df3eb0> #> geospatial_types: [3] #> xmin: 625858.1940030524, xmax: 775539.5804042327 #> ymin: 4733644.250368893, ymax: 4989817.92403143 #> zmin: None, zmax: None #> mmin: None, mmax: None gdf2 = geopandas.GeoDataFrame.from_arrow(f.read()) gdf2.crs.to_authority() #> ('EPSG', '32618') geopandas.testing.assert_geodataframe_equal(gdf2, gdf) ``` * GitHub Issue: #45522 Lead-authored-by: Dewey Dunnington <[email protected]> Co-authored-by: Dewey Dunnington <[email protected]> Co-authored-by: Kristin Cowalcijk <[email protected]> Co-authored-by: Gang Wu <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Dewey Dunnington <[email protected]>

paleolimbot and others added 30 commits August 12, 2024 14:22

update thrift

4a4ebc9

update thrift defs

6f4e4b7

add stubs

6acb2f2

split methods out of line

6ff5855

maybe to/from thrift

0ac5d84

a few more serializers

8a80ac7

add basic test for serialization

f330997

add sort order check

b34b1c2

update thrift

7e9f9b2

start geom utiles

6a9ac3a

test roundtrip thrift cases

a22908f

more geometry utils

dcc083a

more

9acc840

bounder

e18412c

add basic test

76b3f59

a few more strings

fc77ff2

test some bounding box things

21f11a1

more tests

6bbce5c

fix test

bb41b06

with passing tests

88a42f5

add in WKT equiv

3798ef1

more tests

ccbd616

start on stats

3752f83

implement update/merge for geometry statistics

2ff6078

more complete stats

79e4715

start on factory methods

4444757

more stats things

176c997

maybe work with serde

1de193e

Update cpp/src/parquet/types.cc

d2f8157

Co-authored-by: Gang Wu <[email protected]>

Updated parquet.thrift and re-generated cpp sources

b525e84

github-actions bot added the awaiting change review Awaiting change review label Sep 16, 2024

wgtmac reviewed Sep 18, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 18, 2024

wgtmac mentioned this pull request Sep 18, 2024

PARQUET-2471: Add GEOMETRY and GEOGRAPHY logical types apache/parquet-format#240

Merged

Change the sort order of geometry from unknown to unsigned; resolved …

6c322d5

…several review comments

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 19, 2024

Kontinuation added 5 commits September 19, 2024 15:59

Add generate_covering_ member to be explicit that' we'll generate the…

cd43ba5

… coverings from bounding box when populating the encoded statistics

Refactor unscoped enums in geometry_util_internal to enum classes

33803bc

Revert more special case handling for unknown sort order

31a70c5

Fix WKB covering test to take native endianness into consideration

7dfbf4b

min/max of geometry columns are the WKB representation of lower-left …

5d8ab77

…and upper-right points

wgtmac reviewed Sep 19, 2024

View reviewed changes

Kontinuation added 7 commits September 20, 2024 10:38

Address latest review comments

ae5926f

A better implementation of geometry min/max statistics

672f19c

Update the code to accomodate the latest changes of the standard:

26ba162

1. Remove metadata property of geometry logical types 2. Remove covering from geometry statistics

Fix problem decoding WKB geometries with more than 32 coordinates

fe8a3e5

Re-implemented geometry statistics according to the updated spec:

ba80f3e

1. geometry statistics moved out of statistics, it is now a field of column metadata 2. geometry statistics is removed from page index

Merge branch 'main' into kontinuation-parquet-geometry

9aca79d

Revert some unnecessary changes

da55a55

Kontinuation force-pushed the kontinuation-parquet-geometry branch from 8a50947 to da55a55 Compare October 30, 2024 15:01

emkornfield reviewed Feb 6, 2025

View reviewed changes

paleolimbot mentioned this pull request Feb 7, 2025

GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations #45459

Merged

	: LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNKNOWN),
	: LogicalType::Impl(LogicalType::Type::GEOMETRY, SortOrder::UNSIGNED),


		class GeometryStatisticsImpl;

		class PARQUET_EXPORT GeometryStatistics {


		namespace parquet::geometry {

		constexpr double kInf = std::numeric_limits<double>::infinity();

Proof-of-concept Parquet GEOMETRY logical type implementation #43977

Are you sure you want to change the base?

Proof-of-concept Parquet GEOMETRY logical type implementation #43977

Uh oh!

Conversation

Kontinuation commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Kontinuation commented Sep 17, 2024

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kontinuation commented Sep 19, 2024

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Kontinuation commented Sep 6, 2024 •

edited

Loading