Add support for filesystem cache in Store Gateway

[fpetkovski]

Is your proposal related to a problem?

With larger Thanos deployments, memory caching can become impractical due to the size of the data that has to be held in cache.
The cache can quickly fill up, causing items to get evicted on regular basis.

Describe the solution you'd like

I would like to be able to cache chunks on disk in order to take advantage of larger caching space.

Describe alternatives you've considered

None that I can think of.

[hanjm]

+1, I am also request same feature ago. #4638

[GiedriusS]

Sounds good however I think it's worth keeping a few things in mind:

• Ideally the cache wouldn't use mmap due to the reasons outlined here. This means that we shouldn't use bbolt

• What would be the relationship of this cache with the caching bucket?

• What will be the structure of those files?

Perhaps it would be worth putting together a design document?

[fpetkovski]

These are good questions.

What would be the relationship of this cache with the caching bucket?

The way I see it, since FS caching would be slower than memory, it could serve as an L2 cache between memory and S3. So it would be a decorator that implements the existing cache.Cache interface and wraps another cache as fallback.

What will be the structure of those files?

I can think of two different approaches that we can use here. Since we can use the filesystem more liberally, we can periodically download entire chunk files to disk according to a certain strategy. For example, the FS cache might download chunk files from most recent blocks asynchronously and evict chunk files from oldest blocks as it fills up.

Another approach would be to use it the same way we use the memory caches, basically store only individual chunks on disk instead of entire chunk files. For something like maybe we could use an existing KV store like badger or something simpler to avoid having lots of small files.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

I think this makes sense. Are you still interested in working on this? @fpetkovski @hanjm

Another approach would be to use it the same way we use the memory caches, basically store only individual chunks on disk instead of entire chunk files. For something like maybe we could use an existing KV store like badger or something simpler to avoid having lots of small files.

What's the benefits of using a KV over FS? To avoid having small files only?

[fpetkovski]

Unfortunately I don't have time to work on something like this yet, but I would love to see it picked up.

What's the benefits of using a KV over FS? To avoid having small files only?

Yes, that's the main reason. We could also download entire chunk files but I feel it might get pretty wasteful to do that.

[GiedriusS]

Now that I think about this more, I would love to see this implemented as sparse files https://en.m.wikipedia.org/wiki/Sparse_file instead of shoehorning everything into k/v storage systems because they aren't as effective for such use case. For example, there are alignment issues and so on. The only caveat is that some OSes/FS might not support sparse files.

[hanjm]

I think KV over FS, too many small files will leading low performance and maintain difficulty.
I known the clickhouse choosen rocksdb for speed up load many small metadata files.
ClickHouse/ClickHouse#32928

[yeya24]

It would be great to see someone pick this up. I want to extend this usecase to index cache, especially expanded postings and those large postings. It is more efficient to fetch them locally rather than fetch through network which consumes a lot of bandwidth.