test: recordbatch+row ordering

[de-sh]

Fixes #XXXX.

Tests to validate order of recordbatch+row traversal in MergedReverseRecordReader. Improves coverage.

Description

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

Summary by CodeRabbit

• Tests
  • Expanded test coverage for improved validation of data reading features.
  • Introduced new scenarios to simulate varied data batches and handle edge cases such as empty inputs, partial reads, and reverse ordering.
  • These enhancements contribute to a more robust and reliable system performance for end-users.

[coderabbitai]

Walkthrough

The changes enhance the test module in src/parseable/staging/reader.rs by introducing new helper functions and additional test cases. The new helpers facilitate the creation and writing of record batches, while the tests verify the functionality of the OffsetReader and the MergedReverseRecordReader. Updated imports ensure proper module inclusion for file handling, schema definitions, and directory management. These modifications improve test coverage and validate various reading scenarios such as empty offsets, partial reads, and large buffer resizing.

Changes

File	Change Summary
src/parseable/staging/reader.rs	• Added helper functions: create_test_batches and write_test_batches. • Added test cases: test_offset_reader, test_merged_reverse_record_reader, test_empty_offset_list, test_partial_reads, test_get_reverse_reader_single_message, and test_large_buffer_resizing. • Updated imports for file handling and schema management.

Sequence Diagram(s)

sequenceDiagram
    participant TR as Test Runner
    participant HF as Helper Functions
    participant FS as File System
    participant OR as OffsetReader
    participant MR as MergedReverseReader

    TR->>HF: create_test_batches()
    TR->>HF: write_test_batches()
    HF->>FS: Write record batches to file
    TR->>OR: Initiate OffsetReader test
    OR->>TR: Return read bytes/data
    TR->>MR: Initiate reverse reading test
    MR->>TR: Return reverse record batches

Loading

Suggested Reviewers

• nikhilsinhaparseable

Poem

I'm a little rabbit, hopping with delight,
New tests and helpers making code shine bright.
I nibble on offsets, and crunch on records in reverse,
With batches and bytes, each test is a verse.
In this garden of code, I cheer with a joyful hop,
Celebrating smooth changes that never stop!
🐇🌟

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)

• We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
  - To enable this feature, set early_access to true under in the settings.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b4c2bd5 and c9d3a17.

📒 Files selected for processing (1)

• src/parseable/staging/reader.rs (2 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (9)

• GitHub Check: Build Default x86_64-unknown-linux-gnu
• GitHub Check: Build Default aarch64-apple-darwin
• GitHub Check: Build Default aarch64-unknown-linux-gnu
• GitHub Check: Build Default x86_64-pc-windows-msvc
• GitHub Check: Build Default x86_64-apple-darwin
• GitHub Check: Build Kafka x86_64-unknown-linux-gnu
• GitHub Check: Quest Smoke and Load Tests for Distributed deployments
• GitHub Check: Quest Smoke and Load Tests for Standalone deployments
• GitHub Check: coverage

🔇 Additional comments (10)

src/parseable/staging/reader.rs (10)

324-340: LGTM: Imports updated appropriately.

The imports have been expanded to include necessary types and functions required for the new tests, including io operations, path manipulation, and temporary directory support.

---

341-341: Good use of self-import for testing.

Importing the module's own types using a fully qualified path makes it clear that the test is operating on this module's components and avoids potential future issues if the module structure changes.

---

456-477: LGTM: Well-structured test data creation function.

The create_test_batches helper function is well-designed:

• It creates predictable RecordBatch instances with deterministic values
• Batch numbering is reflected in the data values, making test validation straightforward
• Each batch contains two rows for thorough testing
• Error handling is appropriate for a test helper

---

479-495: LGTM: Clean batch writing utility.

The write_test_batches function:

• Provides proper error handling with io::Result return type
• Has good parameter separation (path, schema, batches)
• Correctly finalizes the writer with finish()

---

497-522: LGTM: Comprehensive OffsetReader test.

This test is thorough in validating core OffsetReader functionality:

• Tests with multiple offset sections
• Verifies correct byte sequence is read
• Confirms end-of-read behavior

---

524-584: Good test for MergedReverseRecordReader with detailed assertions.

Excellent test that validates:

• Reverse order batch processing
• Proper row reversal within batches (via value assertions)
• Complete traversal of all batches
• End-of-iteration detection

The test clearly proves that both recordbatch-level and row-level ordering work as expected.

---

586-598: LGTM: Edge case handled correctly.

Good test for the empty offset list case, which verifies the documented safety invariant on line 174 that states "On empty list the reader returns no bytes read."

---

600-625: LGTM: Partial reads test validates buffer handling.

This test correctly verifies that OffsetReader can handle partial reads when:

1. The target buffer is smaller than the data section
2. Multiple reads are needed to complete a section
3. The reader properly tracks position within sections

This ensures robust behavior when clients provide undersized buffers.

---

627-658: LGTM: Single message test validates schema handling.

Good test that verifies the reader works correctly with minimal data (just a single record batch). This is an important edge case, especially for handling schema messages.

---

660-680: LGTM: Buffer resizing test validates adaptive memory usage.

This test verifies that the OffsetReader can dynamically resize its internal buffer to accommodate large data sections, which is important for performance and memory management.

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.