Hierarchical caching for s3fs

[ValHayot]

Hi,

Apologies if this is not the right place to ask this; didn't know where it was best to ask.

I'm a graduate student working on building tools to simplify data management (and improve data transfer performance) for scientific applications. I've been looking at using a cache hierarchy using available local storage media (eg. RAM, SSD, HDD) to speed up writes, as well as perhaps more pipeline-dependent attributes such as prefetching, and data management within the cache (i.e. asynchronous flushing and eviction).

For a research project I was thinking of applying these features to s3fs but wanted to check with you first if it would be useful at all. I know there is a caching mechanism in fsspec and it looks like it can only be to specific path (rather than a multitude of potential paths), but I could be wrong.

Thanks!

[martindurant]

Thanks for the interest!

Whereas s3fs is the most-downloaded addon to fsspec, I do think that the caching layer belongs there rather than here.

You should note that it is already possible to reference multiple (local filesystem) locations in a caching instance, of which the last one is considered to be the only writable one.

A few related open issues that you might want to work on:

• config for caching via fsspec and disable caching filesystem_spec#493
• feature: size limit on caching filesystem filesystem_spec#510
• caching location to be non-local filesystem_spec#494 (this would perhaps be the most interesting, to enable RAM and cluster/cloud caches at either side of the current local filesystem implementation)

[ValHayot]

Awesome! Thanks for the suggestions. I should be able to help out with some of those.
I'll post on the fsspec repo if I have any questions related to the above or similar things.