[core] support btree global index in paimon-common

[steFaiz]

Purpose

This PR is a part of #6834, aiming to provide capability in paimon-common module.

File Format

The introduced BTree Index File is also base on SST File Format, with slight difference with LookupStore File, as illustrated below:

The file is composed by:

1. A normal sst file
2. A null bitmap
3. A Footer containing all handles and magic number

File Entry

The key-value pair of internal SST File is as below:

• key: the serialized key bytes
• value: a list of row id, which is encoded as a varLenInt and following varLenLongs.

Tests

Please see:

• org.apache.paimon.globalindex.btree.BTreeFileMetaSelectorTest
• org.apache.paimon.globalindex.btree.BTreeGlobalIndexerTest

API and Format

None

Documentation

Will be added in the future.

[steFaiz]

Test failures are irrelevant to this PR. Will try to reopen after all reviews addressed.

[JingsongLi]

@steFaiz Can you open actions in your repo too? This allows for double verification.

[steFaiz]

@JingsongLi Thanks! I've enabled all workflows in my forked repo.

[JingsongLi]

I think we can just introduce another format.
Lookup Sst file:

 *     +-----------------------------------+------+
 *     |             Footer                |      |
 *     +-----------------------------------+      |
 *     |           Index Block             |      +--> Loaded on open
 *     +-----------------------------------+      |
 *     |        Bloom Filter Block         |   +-->  Loaded on first Lookup, memory can be expired
 *     +-----------------------------------+------+
 *     |            Data Block             |      |
 *     +-----------------------------------+      |
 *     |              ......               |      +--> Loaded on requested
 *     +-----------------------------------+      |
 *     |            Data Block             |      |
 *     +-----------------------------------+------+

BTree Index File:

 *     +-----------------------------------+------+
 *     |             Footer                |      |
 *     +-----------------------------------+      |
 *     |           Index Block             |      +--> Loaded on open
 *     +-----------------------------------+      |
 *     |        Bloom Filter Block       |      +--> Loaded on first Lookup, memory can be expired
 *     +-----------------------------------+      |
 *     |         Null bitmap Block    |      +--> Loaded on requested
 *     +-----------------------------------+------+
 *     |            Data Block             |      |
 *     +-----------------------------------+      |
 *     |              ......               |      +--> Loaded on requested
 *     +-----------------------------------+      |
 *     |            Data Block             |      |
 *     +-----------------------------------+------+

[steFaiz]

I think we can just introduce another format. Lookup Sst file:

Thanks for your inspiration! I will try to refactor the code!

[JingsongLi]

Looks good to me!