Column indexes

[asfimport]

Write the column indexes described in PARQUET-922.
This is the first phase of implementing the whole feature. The implementation is done in the following steps:

• Utility to read/write indexes in parquet-format

• Writing indexes in the parquet file

• Extend parquet-tools and parquet-cli to show the indexes

• Limit index size based on parquet properties

• Trim min/max values where possible based on parquet properties

• Filtering based on column indexes

  The work is done on the feature branch column-indexes. This JIRA will be resolved after the branch has been merged to master.

Reporter: Gabor Szadovszky / @gszadovszky
Assignee: Gabor Szadovszky / @gszadovszky

Subtasks:

• Column indexes: read/write API
• Column indexes: Show indexes in tools
• Column indexes: Limit index size
• Column indexes: Truncate min/max values
• Column indexes: Filtering
• Column Indexes: Invalid row indexes for pages starting with nulls
• Incorrect check for ASCENDING/DESCENDING at column index write path
• Fix issues of NaN and +-0.0 in case of float/double column indexes
• Improve value skipping at page synchronization
• appendRowGroup will loose pageIndex

Related issues:

• Don't write page level statistics (blocks)
• Write index page in parquet file (is duplicated by)
• Limit page size based on maximum row count (relates to)
• Make Spark SQL support Column indexes (relates to)
• Add index pages to the format to support efficient page skipping (depends upon)
• Improve logic when to write column indexes (is depended upon by)
• Benchmark filtering column-indexes (is depended upon by)

PRs and other links:

• GitHub Pull Request #527
• format PR

_{Note: This issue was originally created as PARQUET-1201. Please see the migration documentation for further details.}

[asfimport]

Gabor Szadovszky / @gszadovszky:
@rdblue, I've added a pull request for parquet-format which is required for the changes to be done in parquet-mr. I guess, we also shall create a parquet-format release with this change so we can rely on these changes in parquet-mr. Could you please help in this?

[asfimport]

Pankaj Sharma:
Thanks for all the work on this 1201.  Cant wait for it to get completed so we can utilize secondary indexes in Parquet

[asfimport]

Gabor Szadovszky / @gszadovszky:
Thanks, [~pshar0]. I'm waiting for some reviews for the last task. If it is ready I'll push it to the feature branch and start working on merging the whole branch to master.

Hopefully, will be done soon.

[asfimport]

Ryan Blue / @rdblue:
@gszadovszky, where is the branch for page skipping? Is it this one? https://github.com/apache/parquet-mr/tree/column-indexes

I just went to review it, but I don't see a PR. Could you open one against master?

[asfimport]

Zoltan Ivanfi / @zivanfi:
@rdblue That is the branch indeed. Our idea with the feature branch was that each commit to the feature branch will happen via a PR and will go through the regular thorough reviewing process. We wanted to avoid opening a giant PR at the end which would be very hard to review anyway due to its sheer size. The community accepted this approach on the Parquet sync where we discussed it. Could you please review the individual PR-s instead? Thanks!