Skip to content

Commit 462188c

Browse files
Merge main into claude/issue-161-20251029-0140
Resolved conflicts between S3 PR and FileTreeStore addition: - Added both S3 and FileTree store to documentation - Kept both S3 and filetree dependencies in pyproject.toml - Updated docs to mention both stores Co-authored-by: William Easton <[email protected]>
2 parents 4a231c5 + 3b3def3 commit 462188c

File tree

7 files changed

+635
-22
lines changed

7 files changed

+635
-22
lines changed

docs/api/stores.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,16 @@ Persistent disk-based key-value store using DiskCache.
2323
members:
2424
- __init__
2525

26+
## FileTree Store
27+
28+
Directory-based store for visual inspection and testing.
29+
30+
::: key_value.aio.stores.filetree.FileTreeStore
31+
options:
32+
show_source: false
33+
members:
34+
- __init__
35+
2636
## Redis Store
2737

2838
Redis-backed key-value store.
@@ -43,6 +53,16 @@ AWS DynamoDB-backed key-value store.
4353
members:
4454
- __init__
4555

56+
## S3 Store
57+
58+
AWS S3-backed key-value store.
59+
60+
::: key_value.aio.stores.s3.S3Store
61+
options:
62+
show_source: false
63+
members:
64+
- __init__
65+
4666
## Elasticsearch Store
4767

4868
Elasticsearch-backed key-value store.

docs/stores.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Local stores are stored in memory or on disk, local to the application.
3434
| Memory | N/A ||| Fast in-memory storage for development and caching |
3535
| Disk | Stable | ☑️ || Persistent file-based storage in a single file |
3636
| Disk (Per-Collection) | Stable | ☑️ || Persistent storage with separate files per collection |
37+
| FileTree (test) | Unstable | ☑️ || Directory-based storage with JSON files for visual inspection |
3738
| Null (test) | N/A ||| No-op store for testing without side effects |
3839
| RocksDB | Unstable | ☑️ || High-performance embedded database |
3940
| Simple (test) | N/A ||| Simple in-memory store for testing |
@@ -140,6 +141,74 @@ pip install py-key-value-aio[disk]
140141

141142
---
142143

144+
### FileTreeStore
145+
146+
Directory-based storage for visual inspection and debugging.
147+
148+
```python
149+
from key_value.aio.stores.filetree import FileTreeStore
150+
151+
store = FileTreeStore(directory="./debug-store")
152+
```
153+
154+
**Installation:**
155+
156+
```bash
157+
pip install py-key-value-aio[filetree]
158+
```
159+
160+
**Use Cases:**
161+
162+
- Visual inspection of store contents
163+
- Debugging store behavior
164+
- Development and testing
165+
- Understanding data structure
166+
167+
**Characteristics:**
168+
169+
- Collections as directories
170+
- Keys as JSON files (`{key}.json`)
171+
- Human-readable filesystem layout
172+
- Easy to inspect and modify
173+
- **NOT for production use**
174+
175+
**Directory Structure:**
176+
177+
```text
178+
{base_directory}/
179+
{collection_1}/
180+
{key_1}.json
181+
{key_2}.json
182+
{collection_2}/
183+
{key_3}.json
184+
```
185+
186+
**Important Limitations:**
187+
188+
- Poor performance with many keys
189+
- No atomic operations
190+
- No automatic cleanup of expired entries
191+
- Filesystem path length constraints
192+
- Subject to filesystem limitations
193+
194+
**When to Use:**
195+
196+
Use FileTreeStore when you need to:
197+
198+
- Visually inspect what's being stored
199+
- Debug complex data structures
200+
- Understand how the store organizes data
201+
- Manually modify stored data for testing
202+
203+
**When NOT to Use:**
204+
205+
- Production environments
206+
- High-performance requirements
207+
- Large datasets
208+
- Concurrent access scenarios
209+
210+
---
211+
143212
### RocksDBStore
144213

145214
High-performance embedded database using RocksDB.
@@ -328,6 +397,7 @@ Distributed stores provide network-based storage for multi-node applications.
328397
| Store | Stability | Async | Sync | Description |
329398
|-------|:---------:|:-----:|:----:|:------------|
330399
| DynamoDB | Unstable || ✖️ | AWS DynamoDB key-value storage |
400+
| S3 | Unstable || ✖️ | AWS S3 object storage |
331401
| Elasticsearch | Unstable ||| Full-text search with key-value capabilities |
332402
| Memcached | Unstable || ✖️ | High-performance distributed memory cache |
333403
| MongoDB | Unstable ||| Document database used as key-value store |
@@ -434,6 +504,42 @@ pip install py-key-value-aio[dynamodb]
434504

435505
---
436506

507+
### S3Store
508+
509+
AWS S3 object storage for durable, scalable key-value storage.
510+
511+
```python
512+
from key_value.aio.stores.s3 import S3Store
513+
514+
store = S3Store(
515+
bucket_name="my-kv-bucket",
516+
region_name="us-east-1"
517+
)
518+
```
519+
520+
**Installation:**
521+
522+
```bash
523+
pip install py-key-value-aio[s3]
524+
```
525+
526+
**Use Cases:**
527+
528+
- Large value storage (up to 5TB per object)
529+
- Durable, long-term storage
530+
- Cost-effective archival
531+
- Multi-region replication
532+
533+
**Characteristics:**
534+
535+
- 99.999999999% durability
536+
- Automatic key sanitization for S3 path limits
537+
- Supports lifecycle policies
538+
- Pay-per-use pricing
539+
- Stable storage format: **Unstable**
540+
541+
---
542+
437543
### ElasticsearchStore
438544

439545
Full-text search engine used as a key-value store.

key-value/key-value-aio/pyproject.toml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ py-key-value-shared-test = { workspace = true }
3434
[project.optional-dependencies]
3535
memory = ["cachetools>=5.0.0"]
3636
disk = ["diskcache>=5.0.0", "pathvalidate>=3.3.1",]
37+
filetree = ["aiofile>=3.5.0", "anyio>=4.4.0"]
3738
redis = ["redis>=4.3.0"]
3839
mongodb = ["pymongo>=4.0.0"]
3940
valkey = ["valkey-glide>=2.1.0"]
@@ -68,7 +69,7 @@ env_files = [".env"]
6869

6970
[dependency-groups]
7071
dev = [
71-
"py-key-value-aio[memory,disk,redis,elasticsearch,memcached,mongodb,vault,dynamodb,s3,rocksdb]",
72+
"py-key-value-aio[memory,disk,filetree,redis,elasticsearch,memcached,mongodb,vault,dynamodb,s3,rocksdb]",
7273
"py-key-value-aio[valkey]; platform_system != 'Windows'",
7374
"py-key-value-aio[keyring]",
7475
"py-key-value-aio[pydantic]",
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
"""File-tree based store for visual inspection and testing."""
2+
3+
from key_value.aio.stores.filetree.store import (
4+
FileTreeStore,
5+
FileTreeV1CollectionSanitizationStrategy,
6+
FileTreeV1KeySanitizationStrategy,
7+
)
8+
9+
__all__ = [
10+
"FileTreeStore",
11+
"FileTreeV1CollectionSanitizationStrategy",
12+
"FileTreeV1KeySanitizationStrategy",
13+
]

0 commit comments

Comments
 (0)