Identify more base store shared utilities

[strawgate]

We should see if there's any code reduction/simplification we can provide by sharing more code across stores or wrappers

[claude]

Claude finished @strawgate's task —— View job

---

Investigation Complete: Shared Utilities Analysis ✓

I've conducted a thorough analysis of the py-key-value codebase to identify opportunities for reducing code duplication and simplifying implementations by sharing more code across stores and wrappers.

---

Recommendation

Implement 7 new shared utilities and 2 base class enhancements to eliminate ~500+ lines of duplicated code across 15+ store implementations and simplify future store development. The highest-impact opportunities are:

1. Batch operation guard helper (eliminates 30+ duplicate if not keys: return [] checks)
2. Client initialization helper (consolidates 5 different URL/client patterns)
3. Collection/key sanitization mixins (shares logic across 4 stores)
4. Custom serialization adapter base class (eliminates duplication in MongoDB/Elasticsearch)
5. Bulk operation builder utilities (shares patterns across Redis, Elasticsearch, MongoDB)

---

Findings

After analyzing all store implementations, wrapper patterns, and shared utilities, I identified seven major categories of code duplication with varying complexity and impact:

1. Empty Sequence Guards (High Impact, Low Complexity)

Pattern Found: All stores implementing _get_managed_entries, _put_managed_entries, and _delete_managed_entries include identical empty-check guards:

if not keys:
    return []  # or return 0 for delete

Current Occurrences:

• Redis: 3 occurrences (get_many, put_many, delete_many)
• Valkey: 3 occurrences
• MongoDB: 3 occurrences
• Elasticsearch: 3 occurrences
• Memcached: 1 occurrence
• RocksDB: 1 occurrence

Total: 14+ duplicate implementations

Opportunity: Create a decorator or helper function in key-value-shared to eliminate this boilerplate.

# Proposed shared utility
from functools import wraps

def guard_empty_sequence(return_value):
    """Decorator that returns early if keys/values sequence is empty."""
    def decorator(func):
        @wraps(func)
        async def wrapper(self, *, keys: Sequence[str], **kwargs):
            if not keys:
                return return_value
            return await func(self, keys=keys, **kwargs)
        return wrapper
    return decorator

2. Client Initialization Patterns (High Impact, Medium Complexity)

Pattern Found: Multiple stores support three initialization patterns with duplicated logic:

1. Accept existing client
2. Accept URL string
3. Accept individual connection parameters (host, port, etc.)

Current Implementations:

• Redis (redis/store.py:42-88): urlparse logic for Redis URLs
• MongoDB (mongodb/store.py:139-177): client vs URL selection
• Elasticsearch (elasticsearch/store.py:171-214): client vs URL+API key
• Vault (vault/store.py:52-71): similar pattern
• DynamoDB (dynamodb/store.py:78-116): session creation pattern

Code Duplication: Each store has 20-40 lines of similar conditional logic

Opportunity: Create base client initialization helpers:

# Proposed shared utility in key-value-shared
class ClientOrURLMixin:
    """Mixin for stores that support both client and URL initialization."""
    
    @staticmethod
    def resolve_client_or_url(
        client: T | None,
        url: str | None, 
        client_factory: Callable[[str], T],
        error_message: str
    ) -> T:
        if client:
            return client
        elif url:
            return client_factory(url)
        else:
            raise ValueError(error_message)

3. Collection/Key Sanitization (Medium Impact, Medium Complexity)

Pattern Found: Four stores have very similar sanitization patterns but implement them separately:

• MongoDB (mongodb/store.py:190-203): _sanitize_collection_name() with 200 char limit, alphanumeric+underscore
• Elasticsearch (elasticsearch/store.py:232-246): _sanitize_index_name() with 240 char limit, lowercase+numbers+special chars
• Keyring (keyring/store.py): Similar patterns
• Windows Registry (windows_registry/store.py): Similar patterns

Current Shared Utility: sanitize_string() exists in key-value-shared/src/key_value/shared/utils/sanitize.py but stores duplicate the method creation

Opportunity: Create sanitization mixin or base methods that stores can inherit:

# Enhanced base class support
class BaseSanitizedStore(BaseStore):
    """Base class for stores requiring sanitized collection/key names."""
    
    # Subclasses override these
    _collection_max_length: int
    _collection_allowed_chars: str
    _key_max_length: int  
    _key_allowed_chars: str
    
    def _sanitize_collection(self, collection: str) -> str:
        return sanitize_string(
            value=collection,
            max_length=self._collection_max_length,
            allowed_characters=self._collection_allowed_chars
        )

4. Custom Serialization Adapters (Medium Impact, High Complexity)

Pattern Found: MongoDB and Elasticsearch both create custom SerializationAdapter subclasses with nearly identical patterns:

MongoDB (mongodb/store.py:39-89):

class MongoDBSerializationAdapter(SerializationAdapter):
    def prepare_dump(self, data: dict[str, Any]) -> dict[str, Any]:
        value = data.pop("value")
        data["value"] = {}
        if self._native_storage:
            data["value"]["object"] = value
        else:
            data["value"]["string"] = value
        return data

Elasticsearch (elasticsearch/store.py:89-133):

class ElasticsearchSerializationAdapter(SerializationAdapter):
    def prepare_dump(self, data: dict[str, Any]) -> dict[str, Any]:
        value = data.pop("value")
        data["value"] = {}
        if self._native_storage:
            data["value"]["flattened"] = value
        else:
            data["value"]["string"] = value
        return data

Opportunity: Create a configurable base adapter class:

class NativeStorageAdapter(SerializationAdapter):
    """Adapter supporting both native and string storage modes."""
    
    def __init__(
        self,
        native_storage: bool = True,
        native_field_name: str = "object",
        string_field_name: str = "string"
    ):
        # ... implementation

5. Compound Key Construction (Low Impact - Already Well Abstracted)

Current State: ✅ Well abstracted in key-value-shared/src/key_value/shared/utils/compound.py

The following utilities are already shared and working well:

• compound_key() - Used by Redis, Valkey, Memcached, RocksDB
• get_keys_from_compound_keys() - Used for key enumeration
• prefix_key() / unprefix_key() - Used by PrefixKeysWrapper
• prefix_collection() / unprefix_collection() - Used by PrefixCollectionsWrapper

No action needed - this is a good example of effective code sharing.

6. Bulk Operation Builders (Medium Impact, High Complexity)

Pattern Found: Multiple stores build bulk operation structures differently but with similar patterns:

• Redis (redis/store.py:146-182): Pipeline pattern for setex with TTL
• MongoDB (mongodb/store.py:290-323): UpdateOne operations list
• Elasticsearch (elasticsearch/store.py:346-378): Bulk action list with alternating metadata/document

Current Duplication: Each store has 20-40 lines building operation lists

Opportunity: While the backends differ significantly, there's potential for helper functions:

# Proposed utility
def build_bulk_operations(
    keys: Sequence[str],
    values: Sequence[T],
    operation_builder: Callable[[str, T], Operation]
) -> list[Operation]:
    """Build bulk operations from keys and values."""
    return [
        operation_builder(key, value) 
        for key, value in zip(keys, values, strict=True)
    ]

Note: This may be too abstract to provide real value. Needs careful consideration.

7. Pagination/Limiting Constants (Low Impact, Low Complexity)

Pattern Found: Multiple stores define identical pagination constants:

DEFAULT_PAGE_SIZE = 10000
PAGE_LIMIT = 10000

Current Occurrences:

• Redis: lines 21-22
• MongoDB: lines 27-28
• Elasticsearch: lines 79-80

Opportunity: Move to shared constants:

# In key-value-shared/src/key_value/shared/constants.py
DEFAULT_ENUMERATION_PAGE_SIZE = 10000
MAX_ENUMERATION_PAGE_SIZE = 10000

---

Detailed Action Plan

Phase 1: Low-Hanging Fruit (High Impact, Low Complexity)

Task 1.1: Add Empty Sequence Guard Decorator

• Create @guard_empty_sequence() decorator in key-value-shared/src/key_value/shared/utils/decorators.py
• Add tests for decorator behavior
• Apply to all _get_managed_entries, _put_managed_entries, _delete_managed_entries implementations
• Run full test suite to verify behavior unchanged

Estimated Impact: Eliminate 14+ duplicate guard clauses

Task 1.2: Centralize Pagination Constants

• Add pagination constants to key-value-shared/src/key_value/shared/constants.py
• Update Redis, MongoDB, Elasticsearch to use shared constants
• Verify no behavioral changes

Estimated Impact: Eliminate 6+ duplicate constant definitions

Phase 2: Client Initialization (High Impact, Medium Complexity)

Task 2.1: Create Client/URL Resolution Helper

• Add ClientOrURLMixin or helper function to key-value-shared
• Document the pattern with examples
• Refactor Redis store to use helper (pilot implementation)
• Verify Redis tests pass
• Roll out to MongoDB, Elasticsearch, Vault stores

Estimated Impact: Reduce client initialization code by ~100 lines across 5 stores

Phase 3: Sanitization Improvements (Medium Impact, Medium Complexity)

Task 3.1: Create Sanitization Mixin or Enhanced Base Class Methods

• Design BaseSanitizedStore mixin or add methods to BaseStore
• Migrate MongoDB to use new pattern (pilot)
• Verify MongoDB tests pass
• Roll out to Elasticsearch, Keyring, Windows Registry

Estimated Impact: Reduce sanitization code by ~60 lines across 4 stores

Phase 4: Serialization Adapters (Medium Impact, High Complexity)

Task 4.1: Create Configurable Native Storage Adapter

• Design NativeStorageAdapter base class
• Migrate MongoDB to use new adapter (pilot)
• Verify all MongoDB tests pass (both native and string storage modes)
• Migrate Elasticsearch to use new adapter
• Verify all Elasticsearch tests pass

Estimated Impact: Reduce adapter code by ~80 lines across 2 stores

Phase 5: Documentation and Future Guidance

Task 5.1: Update AGENTS.md and DEVELOPING.md

• Document new shared utilities
• Add "Implementing a New Store" guide referencing shared utilities
• Update code review checklist to catch duplication opportunities

---

Related Items

Related Issues and Pull Requests

Repository	Issue/PR	Relevance
strawgate/py-key-value	#133: Identify Refactor Opportunities	Previous investigation focused on wrapper refactoring; recommended helper-based transformation system in BaseWrapper
strawgate/py-key-value	#165: Review ManagedEntry	Related refactoring around ManagedEntry and serialization; introduced adapter pattern for store-specific serialization
strawgate/py-key-value	#174: Refactor ManagedEntry serialization	Implemented SerializationAdapter pattern that MongoDB and Elasticsearch now extend

Related Files

Repository	File	Relevance	Key Lines
strawgate/py-key-value	key-value/key-value-aio/src/key_value/aio/stores/base.py	BaseStore class - good foundation for adding shared functionality	180-183, 267-292
strawgate/py-key-value	key-value/key-value-shared/src/key_value/shared/utils/sanitize.py	Existing sanitization utilities that could be better leveraged	64-136
strawgate/py-key-value	key-value/key-value-shared/src/key_value/shared/utils/compound.py	Excellent example of well-abstracted shared utilities	34-82
strawgate/py-key-value	key-value/key-value-aio/src/key_value/aio/stores/redis/store.py	Representative store showing common patterns	42-88, 105-123
strawgate/py-key-value	key-value/key-value-aio/src/key_value/aio/stores/mongodb/store.py	Custom serialization adapter pattern	39-89, 190-203
strawgate/py-key-value	key-value/key-value-aio/src/key_value/aio/stores/elasticsearch/store.py	Custom serialization adapter pattern (similar to MongoDB)	89-133, 232-246

---

Problems Encountered During Investigation

None - all investigation tools and codebase resources were accessible and provided comprehensive information for the analysis.

---