[Feature] Introduce SST File Format base on SortLookupStore

[steFaiz]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The SST file format can be used in the following scenarios:

1. As the local storage format for LookupFile.
2. In the B-tree global index.
3. When query performance is a higher priority—for example, when used as a Flink dimension table—the data can be directly stored in SST format.

Solution

Based on current LookupStore implementation, we can introduce a generalized SST FileFormat which can store any binary key-value pairs. The file format is expected to provide below functionality：

1. fast lookup
2. fast range query
3. fast random access
4. efficient full-file scan

Anything else?

Please refer to pip-25 as the related proposal and refer to PR as current draft implementation.

Are you willing to submit a PR?

• I'm willing to submit a PR!