Skip to content

Conversation

dbarbashov
Copy link
Contributor

While I was trying to find a reason why s3 store started losing blocks (ipld block not found error). I've played around with the repo and it seems that after few guided vibe-coding iterations it finally became stable. There were several issues with pagination/key handling logic which were addressed.

Potentially this fixes #261. At least it fixed it for me on my node

Key changes:

  • Correctness Fixes:

    • Query Implementation: The Query method has been completely rewritten to be a stateful iterator.
      • It now correctly handles pagination for prefixes that span multiple S3 ListObjectsV2 pages. The previous implementation was limited to a single page of results (max 1000 objects).
      • Offset and Limit are now properly supported across paginated results.
    • Batch Operations: Batch Delete operations now correctly prefix keys with the configured RootDirectory before sending them to S3.
    • Key Path Handling: Key paths are now more robustly handled by consistently trimming leading slashes, aligning with go-datastore conventions.
  • Refactoring & Testability:

    • S3 Interface: The S3 client is now abstracted behind the s3iface.S3API interface.
    • New Unit Tests:
      • Batch operations (Put and Delete)
      • Query pagination, Offset, and Limit
      • Error handling and delete idempotency
  • Observability & Dependencies:

    • Comprehensive Logging: Added structured logging via go-log/v2 to all key datastore operations (Put, Get, Query, Batch, etc.).
    • Dependency Updates: Updated all Go module dependencies to their latest versions, including kubo and go-datastore, and bumped the Go version to 1.25.
    • Housekeeping: Updated .gitignore with common editor and OS patterns.

@lidel lidel requested a review from gammazero September 30, 2025 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IPFS GC not working with the s3.

1 participant