Page-level CRC checksum verification for DataPageV1

[asfimport]

Now that the specification with respect to page level checksums has been clarified (see https://jira.apache.org/jira/browse/PARQUET-1539), we can actually implement it. What needs to be done:

• Implement calculating and writing checksums on write path
• Implement checksum validation on read path
• Make writing out checksums and validation on reads optional (default off)
• Implement tests

Reporter: Boudewijn Braams / @bbraams
Assignee: Boudewijn Braams / @bbraams

Related issues:

• Changed the data order after DataFrame reuse (causes)

PRs and other links:

• GitHub Pull Request #647

_{Note: This issue was originally created as PARQUET-1580. Please see the migration documentation for further details.}

[asfimport]

Natang:
Can this be backported to 1.10.1?

[asfimport]

Gabor Szadovszky / @gszadovszky:
[~natang.salgia], I would rather not backport it to 1.10.1. This is an improvement/feature not a bug fix. We need add the required changes only into a patch release so we have a lower risk that we introduce a regression.

[asfimport]

Steve Loughran / @steveloughran:
My IDE is flagging Checksum.update(byte[]) as since java9, though it is clearly there in openjdk "1.8.0_362". It may be one of those changes backported to some java8 releases but not all....