Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
da86ea2
Add AWS S3 support for py-key-value
github-actions[bot] Oct 29, 2025
1b1c3d7
fix: exclude S3 from sync library and improve test configuration
github-actions[bot] Oct 29, 2025
a8a093f
fix: address CodeRabbit review feedback for S3 store
github-actions[bot] Oct 29, 2025
401d947
fix: handle S3 key length limits by hashing long collection/key names
github-actions[bot] Oct 30, 2025
82d3f6a
refactor: update S3Store to use new serialization adapter pattern
github-actions[bot] Oct 31, 2025
d9c1d7b
feat: add length_is_bytes parameter to sanitize_string and hash_exces…
github-actions[bot] Nov 2, 2025
731afa1
fix: address review feedback for S3 store
github-actions[bot] Nov 2, 2025
d528933
fix: address CodeRabbit review feedback for S3 store
github-actions[bot] Nov 2, 2025
e0f4529
simplify setup/teardown in s3
strawgate Nov 2, 2025
065b551
refactor: align S3 store with sanitization strategy pattern
github-actions[bot] Nov 7, 2025
06228f0
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 7, 2025
dbab902
Longer wait for s3 spinup
strawgate Nov 7, 2025
e5125c2
Small test changes
strawgate Nov 7, 2025
f8b3a70
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 7, 2025
2626137
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 8, 2025
89ccb19
fix: resolve CodeRabbit feedback and failing tests
github-actions[bot] Nov 8, 2025
4a231c5
Merge branch 'main' into claude/issue-161-20251029-0140
strawgate Nov 9, 2025
462188c
Merge main into claude/issue-161-20251029-0140
github-actions[bot] Nov 9, 2025
8618596
Update lockfile and generate sync code for filetree store
github-actions[bot] Nov 9, 2025
f2721ae
Exclude FileTreeStore from sync code generation
github-actions[bot] Nov 9, 2025
f32a8ce
Apply suggestion from @coderabbitai[bot]
strawgate Nov 9, 2025
b230647
Fix Elasticsearch index already exists exception check
github-actions[bot] Nov 9, 2025
5df62e3
Extend Elasticsearch cluster health check timeout from 10s to 15s
github-actions[bot] Nov 9, 2025
ab6c55d
Merge main into claude/issue-161-20251029-0140
github-actions[bot] Nov 9, 2025
f14500a
Remove excessive code comments from S3 store
github-actions[bot] Nov 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This monorepo contains two libraries:

## Why use this library?

- **Multiple backends**: DynamoDB, Elasticsearch, Memcached, MongoDB, Redis,
- **Multiple backends**: DynamoDB, S3, Elasticsearch, Memcached, MongoDB, Redis,
RocksDB, Valkey, and In-memory, Disk, etc
- **TTL support**: Automatic expiration handling across all store types
- **Type-safe**: Full type hints with Protocol-based interfaces
Expand Down Expand Up @@ -131,6 +131,7 @@ pip install py-key-value-aio
pip install py-key-value-aio[memory]
pip install py-key-value-aio[disk]
pip install py-key-value-aio[dynamodb]
pip install py-key-value-aio[s3]
pip install py-key-value-aio[elasticsearch]
# or: redis, mongodb, memcached, valkey, vault, registry, rocksdb, see below for all options
```
Expand Down Expand Up @@ -191,7 +192,7 @@ categories:
- **Local stores**: In-memory and disk-based storage (Memory, Disk, RocksDB, etc.)
- **Secret stores**: Secure OS-level storage for sensitive data (Keyring, Vault)
- **Distributed stores**: Network-based storage for multi-node apps (Redis,
DynamoDB, MongoDB, etc.)
DynamoDB, S3, MongoDB, etc.)

Each store has a **stability rating** indicating likelihood of
backwards-incompatible changes. Stable stores (Redis, Valkey, Disk, Keyring)
Expand Down
10 changes: 10 additions & 0 deletions docs/api/stores.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,16 @@ AWS DynamoDB-backed key-value store.
members:
- __init__

## S3 Store

AWS S3-backed key-value store.

::: key_value.aio.stores.s3.S3Store
options:
show_source: false
members:
- __init__

## Elasticsearch Store

Elasticsearch-backed key-value store.
Expand Down
37 changes: 37 additions & 0 deletions docs/stores.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,7 @@ Distributed stores provide network-based storage for multi-node applications.
| Store | Stability | Async | Sync | Description |
|-------|:---------:|:-----:|:----:|:------------|
| DynamoDB | Unstable | ✅ | ✖️ | AWS DynamoDB key-value storage |
| S3 | Unstable | ✅ | ✖️ | AWS S3 object storage |
| Elasticsearch | Unstable | ✅ | ✅ | Full-text search with key-value capabilities |
| Memcached | Unstable | ✅ | ✖️ | High-performance distributed memory cache |
| MongoDB | Unstable | ✅ | ✅ | Document database used as key-value store |
Expand Down Expand Up @@ -503,6 +504,42 @@ pip install py-key-value-aio[dynamodb]

---

### S3Store

AWS S3 object storage for durable, scalable key-value storage.

```python
from key_value.aio.stores.s3 import S3Store

store = S3Store(
bucket_name="my-kv-bucket",
region_name="us-east-1"
)
```

**Installation:**

```bash
pip install py-key-value-aio[s3]
```

**Use Cases:**

- Large value storage (up to 5TB per object)
- Durable, long-term storage
- Cost-effective archival
- Multi-region replication

**Characteristics:**

- 99.999999999% durability
- Automatic key sanitization for S3 path limits
- Supports lifecycle policies
- Pay-per-use pricing
- Stable storage format: **Unstable**

---

### ElasticsearchStore

Full-text search engine used as a key-value store.
Expand Down
3 changes: 2 additions & 1 deletion key-value/key-value-aio/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ vault = ["hvac>=2.3.0", "types-hvac>=2.3.0"]
memcached = ["aiomcache>=0.8.0"]
elasticsearch = ["elasticsearch>=8.0.0", "aiohttp>=3.12"]
dynamodb = ["aioboto3>=13.3.0", "types-aiobotocore-dynamodb>=2.16.0"]
s3 = ["aioboto3>=13.3.0", "types-aiobotocore-s3>=2.16.0"]
keyring = ["keyring>=25.6.0"]
keyring-linux = ["keyring>=25.6.0", "dbus-python>=1.4.0"]
pydantic = ["pydantic>=2.11.9"]
Expand Down Expand Up @@ -69,7 +70,7 @@ env_files = [".env"]

[dependency-groups]
dev = [
"py-key-value-aio[memory,disk,filetree,redis,elasticsearch,memcached,mongodb,vault,dynamodb,rocksdb,duckdb]",
"py-key-value-aio[memory,disk,filetree,redis,elasticsearch,memcached,mongodb,vault,dynamodb,s3,rocksdb,duckdb]",
"py-key-value-aio[valkey]; platform_system != 'Windows'",
"py-key-value-aio[keyring]",
"py-key-value-aio[pydantic]",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ async def _setup_collection(self, *, collection: str) -> None:
try:
_ = await self._client.options(ignore_status=404).indices.create(index=index_name, mappings=DEFAULT_MAPPING, settings={})
except BadRequestError as e:
if "index_already_exists_exception" in str(e).lower():
if "already_exists_exception" in str(e).lower():
return
raise

Expand Down
13 changes: 13 additions & 0 deletions key-value/key-value-aio/src/key_value/aio/stores/s3/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
"""AWS S3-based key-value store."""

from key_value.aio.stores.s3.store import (
S3CollectionSanitizationStrategy,
S3KeySanitizationStrategy,
S3Store,
)

__all__ = [
"S3CollectionSanitizationStrategy",
"S3KeySanitizationStrategy",
"S3Store",
]
Loading
Loading