Report / blog on parquet metadata sizes for "large" (1000+) numbers of columns

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There are several proposals for remedying perceived issues with parquet which generally propose new formats. For example Lance V2 and Nimble

One of the technical challenges raised about Parquet is that the metadata is encoded such that the entire footer must be read and decoded prior to reading any data.

As the numer of columns increases, the argument goes, the size of the parquet metadata increases beyond the ~8MB sweet spot for a single object store request as well as requiring substantial CPU to decode

However, my theory is that the reason that parquet metadata is typically so large for schemas with many columns is the embedded min/max statistical values for columns / pages

Describe the solution you'd like
I would like to gather data on parquet footer metadata size as a function of:

1. The number of columns
2. The number of row groups
3. if Statistics are enabled / disabled

And then report this in a blog with some sort of conclusion about how well parquet can handle large schemas

Bonus points if we can also measure in memory size (though this will of course vary from implementation to implementation)

Describe alternatives you've considered

Additional context
Related discussion with @wesm on twitter: https://twitter.com/wesmckinn/status/1790884370603024826

Cited issue apache/arrow#39676

[alamb]

I have hopes that I will find time to work on this next week but am not sure

If anyone knows of prior art in this area (measuring the actual metadata size) it would be great if they linked it

[alamb]

One idea would be to create a dataset with 1000 columns, 2000 columns , 5000 columns and 10000 columns with say 10 row groups of 1M rows

Measure the size of the metadata with

1. default writer settings
2. "minimal" settings (no statistics, disable everything possible)

[westonpace]

10K columns by 10 row groups by 1M rows is 100B values (400GB with int32). I don't think anyone has data like that (this is presumptuous, I am probably wrong).

My experience has been either:

• The files they make are much smaller (and thus not enough or undersized row groups) E.g. financial data where ticker is column.
• The columns are very sparse (and thus a need for better sparse encoding). E.g. feature stores.

[westonpace]

Also, I did do some experimentation here (what feels like a long time ago now but was at least a year ago). Two things I observed:

• The in-memory representation of the parquet metadata seemed way too high, orders of magnitude greater than the on-disk size of the metadata. I assume this was specific to the C++ impl and there was just something I was missing but it was pretty significant even with relatively reasonably-sized files (e.g. 50MiB+ per file).
• The parquet-c++ library (and, to a lesser degree, arrow and arrow datasets) were not written with large #'s of columns in mind. So even if the format itself could work, the implementation ends up being O(N^2) in a lot of places where it could presumably do a lot better.

I realize this second point is kind of what you are trying to show. I'd be interested in measurements of the first point, at least enough to prove I was missing something (or this problem is unique to parquet-c++)

[mapleFU]

I don't have specific data by now. At my use case, usally row-group would be limit by size rather than row-count, and we would disable statistics for most of these column. And only write page index without column index in this case

[alamb]

At my use case, usally row-group would be limit by size rather than row-count, and we would disable statistics for most of these column. And only write page index without column index in this case

Yes, this is what I would expect best practice to be for large schemas (especially if the actual data is sparse as @westonpace says). In general I would expect the value for query engines of min/max statistics in parquet metadata to be quite low for columns where the data isn't sorted or clustered in some way.

Thus, if the default parquet writer settings always write this metadata for all columns maybe we need to adjust the default and docuemntation or something

[tustvold]

It may also be worth pointing out that recently support was added to truncate column statistics, page indexes have always supported this, and spark was actually already doing this for columns statistics, with dubious correctness properties. The net result though, is that even is you do write column statistics for all columns, they need not be very large

[tustvold]

I've also filed #5775 to track something I suggested on the mailing list, and would likely drastically reduce the overheads from metadata parsing.

[tustvold]

So I created a toy benchmark (https://github.com/tustvold/arrow-rs/tree/thrift-bench) of a 10,000 column parquet file, with f32 columns, no statistics and 10 row groups.

Default Config

The file metadata for this comes to a chunky 11MB, and it takes the #5777 thrift decoder on 45ms to parse this thrift payload.

If we drop the column count to 1000, the metadata drops to 1MB and parses in ~3.7ms.

If we also drop the row group count to 1, the metadata drops to 100KB and parses in 460 us.

So at a very rough level the cost is 10 bytes and ~450ns per column chunk

Drop ColumnMetadata

Dropping the ColumnMetadata from ColumnChunk drops the size down to 3.6MB and the parsing speed down to 23ms

Drop ColumnChunk

Dropping the columns drops the size down to 109KB and the parsing speed down to 718 us.

At this point we have effectively dropped everything apart from the schema.

Arrow Schema IPC

Now for comparison, I encoded a similar schema using arrow IPC to a flatbuffer. This came to a still pretty chunky 500KB. Validating the offsets in this flatbuffer takes ~1ms, which is longer than decoding the equivalent data from thrift.

Thoughts

• There isn't anything wrong with using thrift, it is competitive if not faster than flatbuffers (when doing full offset validation)
• The cost of encoding ColumnChunk dominates and is O(num_row_groups * num_columns)
• Having a single row group drops the latency down to 2ms even with 10,000 columns. Even if each column only has a single 1MB page, the file will be >10GB even with a single row group

Phrasing the above differently, assuming single page ColumnChunks of 1MB, we can parse the metadata for a column chunk in 450ns. This means for a very large 10GB file we can still parse the metadata within single-digit milliseconds.

[emkornfield]

Nice work. I do think #5770 (comment) is probably the key, that statistics would make things considerably worse, adjusting defaults across implementations probably make sense. I'm not sure what Java is set at but I think C++ will always write all statistics.

[marcin-krystianc]

10K columns by 10 row groups by 1M rows is 100B values (400GB with int32). I don't think anyone has data like that (this is presumptuous, I am probably wrong).

My experience has been either:

• The files they make are much smaller (and thus not enough or undersized row groups) E.g. financial data where ticker is column.
• The columns are very sparse (and thus a need for better sparse encoding). E.g. feature stores.

Hi, we use parquet files with 100 row groups and 50k columns (and this is after the dataset has been split into many individual parquet files). What is worse, our use case is reading individual row groups and only subset of columns.
That makes the cost of reading the entire metadata footer even higher than cost of reading the actual data (because we read entire footer and then read only tiny subset of the actual data).

To deal with the problem we've implemented a tool that stores an index information in a separate file which allows for reading only a necessary subset of metadata. (https://github.com/G-Research/PalletJack)..

[tustvold]

we use parquet files with 100 row groups and 50k columns (and this is after the dataset has been split into many individual parquet files). What is worse, our use case is reading individual row groups and only subset of columns.

How big are these files? At 1MB per page (the recommended size) this would come to 5TB?!

For extremely sparse data, such as feature stores, I can't help wondering if MapArray or similar would be a better way to encode this. You would lose the ability to do projection pushdown in the traditional sense, but maybe this is ok?