LCORE-817: add new e2e tests for tools endpoint #696
Conversation
WalkthroughAdds two end-to-end test scenarios for the Info "tools" endpoint (success and error) and two step implementations to validate tool count and tool JSON structure for a given provider. Changes
Sequence Diagram(s)sequenceDiagram
participant TR as Test Runner
participant S as API Server (/info/tools)
participant L as llama-stack (backend)
rect rgb(235,245,255)
TR->>S: GET /info/tools
alt llama-stack reachable
S->>L: request tools list
L-->>S: tools JSON (200)
S-->>TR: 200 + tools JSON
TR->>TR: validate status 200
TR->>TR: filter by provider (rag-runtime) and assert count == 2
TR->>TR: validate tool JSON structure vs expected schema
else llama-stack unreachable
S->>L: request tools list
L-->>S: error
S-->>TR: 500 + error body
TR->>TR: validate status 500 and error body
end
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
tests/e2e/features/steps/info.py (1)
119-119: Add descriptive error message to assertion.The assertion should include a message showing both expected and actual counts to aid debugging when tests fail, consistent with assertions elsewhere in this file.
Apply this diff:
- assert len(provider_tools) == count + assert len(provider_tools) == count, f"Expected {count} tools for provider '{provider_name}', but found {len(provider_tools)}"
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/e2e/features/info.feature(1 hunks)tests/e2e/features/steps/info.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
tests/e2e/features/**/*.feature
📄 CodeRabbit inference engine (CLAUDE.md)
Write E2E tests as Gherkin feature files for behave
Files:
tests/e2e/features/info.feature
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: All modules start with descriptive module-level docstrings explaining purpose
Use logger = logging.getLogger(name) for module logging after import logging
Define type aliases at module level for clarity
All functions require docstrings with brief descriptions
Provide complete type annotations for all function parameters and return types
Use typing_extensions.Self in model validators where appropriate
Use modern union syntax (str | int) and Optional[T] or T | None consistently
Function names use snake_case with descriptive, action-oriented prefixes (get_, validate_, check_)
Avoid in-place parameter modification; return new data structures instead of mutating arguments
Use appropriate logging levels: debug, info, warning, error with clear messages
All classes require descriptive docstrings explaining purpose
Class names use PascalCase with conventional suffixes (Configuration, Error/Exception, Resolver, Interface)
Abstract base classes should use abc.ABC and @AbstractMethod for interfaces
Provide complete type annotations for all class attributes
Follow Google Python docstring style for modules, classes, and functions, including Args, Returns, Raises, Attributes sections as needed
Files:
tests/e2e/features/steps/info.py
tests/e2e/features/steps/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
Place behave step definitions under tests/e2e/features/steps/
Files:
tests/e2e/features/steps/info.py
tests/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
tests/**/*.py: Use pytest-mock to create AsyncMock objects for async interactions in tests
Use the shared auth mock constant: MOCK_AUTH = ("mock_user_id", "mock_username", False, "mock_token") in tests
Files:
tests/e2e/features/steps/info.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: build-pr
- GitHub Check: e2e_tests (ci)
🔇 Additional comments (2)
tests/e2e/features/info.feature (1)
113-121: LGTM!The error scenario follows the established pattern for testing unreachable llama-stack services and validates the expected error response correctly.
tests/e2e/features/steps/info.py (1)
3-3: LGTM!The
jsonimport is necessary for parsing expected tool structure fromcontext.textin the new step definition.
radofuchs
force-pushed
the
LCORE_817_tools_e2e_tests
branch
from
e422fc2 to
763cdd1
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
tests/e2e/features/steps/info.py (3)
122-122: Fix typo in step definition.The step decorator contains "he response" instead of "the response". This must match the feature file exactly for Behave to recognize the step.
Apply this diff:
-@then("The body of the response has proper structure for provider {provider_name}") +@then("The body of the response has proper structure for provider {provider_name}")Note: Verify the exact wording in
tests/e2e/features/info.featureto ensure alignment.
141-141: Use more specific error message with provider name.Following the pattern from
check_model_structure(line 48) andcheck_shield_structure(line 86), the error message should include the provider name for better diagnostics.Apply this diff:
- assert provider_tool is not None, "No tool found in response" + assert provider_tool is not None, f"No tool found for provider '{provider_name}'"
150-152: Fix copy-paste error in assertion message.The assertion message incorrectly references "model_type" when the assertion is checking "provider_id". This appears to be a copy-paste error from similar validation code.
Apply this diff:
assert ( provider_tool["provider_id"] == expected_json["provider_id"] - ), f"model_type should be {expected_json["provider_id"]}" + ), f"provider_id should be {expected_json['provider_id']}"Note: Also consider using single quotes inside the f-string to avoid potential quote-nesting issues.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/e2e/features/info.feature(1 hunks)tests/e2e/features/steps/info.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/e2e/features/info.feature
🧰 Additional context used
📓 Path-based instructions (3)
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: All modules start with descriptive module-level docstrings explaining purpose
Use logger = logging.getLogger(name) for module logging after import logging
Define type aliases at module level for clarity
All functions require docstrings with brief descriptions
Provide complete type annotations for all function parameters and return types
Use typing_extensions.Self in model validators where appropriate
Use modern union syntax (str | int) and Optional[T] or T | None consistently
Function names use snake_case with descriptive, action-oriented prefixes (get_, validate_, check_)
Avoid in-place parameter modification; return new data structures instead of mutating arguments
Use appropriate logging levels: debug, info, warning, error with clear messages
All classes require descriptive docstrings explaining purpose
Class names use PascalCase with conventional suffixes (Configuration, Error/Exception, Resolver, Interface)
Abstract base classes should use abc.ABC and @AbstractMethod for interfaces
Provide complete type annotations for all class attributes
Follow Google Python docstring style for modules, classes, and functions, including Args, Returns, Raises, Attributes sections as needed
Files:
tests/e2e/features/steps/info.py
tests/e2e/features/steps/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
Place behave step definitions under tests/e2e/features/steps/
Files:
tests/e2e/features/steps/info.py
tests/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
tests/**/*.py: Use pytest-mock to create AsyncMock objects for async interactions in tests
Use the shared auth mock constant: MOCK_AUTH = ("mock_user_id", "mock_username", False, "mock_token") in tests
Files:
tests/e2e/features/steps/info.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: build-pr
- GitHub Check: e2e_tests (azure)
- GitHub Check: e2e_tests (ci)
🔇 Additional comments (1)
tests/e2e/features/steps/info.py (1)
3-3: LGTM!The
jsonimport is necessary for parsing expected tool structure fromcontext.textin the newcheck_tool_structurefunction.
| if tool["provider_id"] == provider_name: | ||
| provider_tools.append(tool) | ||
|
|
||
| assert len(provider_tools) == count |
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Add descriptive assertion message.
The assertion should include a descriptive error message for better test failure diagnostics, consistent with other assertions in this file (lines 12, 37, 38, 48, 75, 76, 86).
Apply this diff:
- assert len(provider_tools) == count
+ assert len(provider_tools) == count, f"Expected {count} tools for provider '{provider_name}', but found {len(provider_tools)}"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| assert len(provider_tools) == count | |
| assert len(provider_tools) == count, f"Expected {count} tools for provider '{provider_name}', but found {len(provider_tools)}" |
🤖 Prompt for AI Agents
In tests/e2e/features/steps/info.py around line 119, the assertion assert
len(provider_tools) == count lacks a descriptive failure message; update it to
include context and actual vs expected values (e.g., "expected {count} provider
tools, got {len(provider_tools)}") so test failures show clear diagnostics,
matching the style used on lines 12, 37, 38, 48, 75, 76, 86.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
tests/e2e/features/steps/info.py (1)
119-119: Add descriptive assertion message.This issue remains unaddressed from the previous review. The assertion lacks a descriptive error message for better test failure diagnostics, inconsistent with other assertions in this file.
Apply this diff:
- assert len(provider_tools) == count + assert len(provider_tools) == count, f"Expected {count} tools for provider '{provider_name}', but found {len(provider_tools)}"
🧹 Nitpick comments (3)
tests/e2e/features/steps/info.py (3)
104-105: Consider enhancing docstring with Google-style sections.The docstring could be improved to include Args and Returns sections for better documentation clarity, consistent with the project's coding guidelines for Google Python docstring style.
Apply this diff:
-def check_tool_count(context: Context, count: int, provider_name: str) -> None: - """Check that the number of tools for defined provider is correct.""" +def check_tool_count(context: Context, count: int, provider_name: str) -> None: + """Check that the number of tools for defined provider is correct. + + Args: + context: Behave context containing the response. + count: Expected number of tools. + provider_name: Name of the provider to filter tools by. + """Based on coding guidelines.
141-141: Consider making error message more specific.The error message could include the provider name for clearer diagnostics, matching the pattern used in similar functions throughout the file.
Apply this diff:
- assert provider_tool is not None, "No tool found in response" + assert provider_tool is not None, f"No tool found for provider '{provider_name}'"
123-124: Consider enhancing docstring with Google-style sections.The docstring could be improved to include Args and Returns sections for better documentation clarity, consistent with the project's coding guidelines for Google Python docstring style.
Apply this diff:
-def check_tool_structure(context: Context, provider_name: str) -> None: - """Check that the first listed tool for defined provider has the correct structure.""" +def check_tool_structure(context: Context, provider_name: str) -> None: + """Check that the first listed tool for defined provider has the correct structure. + + Args: + context: Behave context containing the response and expected JSON in context.text. + provider_name: Name of the provider to filter tools by. + """Based on coding guidelines.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/e2e/features/info.feature(1 hunks)tests/e2e/features/steps/info.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/e2e/features/info.feature
🧰 Additional context used
📓 Path-based instructions (3)
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: All modules start with descriptive module-level docstrings explaining purpose
Use logger = logging.getLogger(name) for module logging after import logging
Define type aliases at module level for clarity
All functions require docstrings with brief descriptions
Provide complete type annotations for all function parameters and return types
Use typing_extensions.Self in model validators where appropriate
Use modern union syntax (str | int) and Optional[T] or T | None consistently
Function names use snake_case with descriptive, action-oriented prefixes (get_, validate_, check_)
Avoid in-place parameter modification; return new data structures instead of mutating arguments
Use appropriate logging levels: debug, info, warning, error with clear messages
All classes require descriptive docstrings explaining purpose
Class names use PascalCase with conventional suffixes (Configuration, Error/Exception, Resolver, Interface)
Abstract base classes should use abc.ABC and @AbstractMethod for interfaces
Provide complete type annotations for all class attributes
Follow Google Python docstring style for modules, classes, and functions, including Args, Returns, Raises, Attributes sections as needed
Files:
tests/e2e/features/steps/info.py
tests/e2e/features/steps/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
Place behave step definitions under tests/e2e/features/steps/
Files:
tests/e2e/features/steps/info.py
tests/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
tests/**/*.py: Use pytest-mock to create AsyncMock objects for async interactions in tests
Use the shared auth mock constant: MOCK_AUTH = ("mock_user_id", "mock_username", False, "mock_token") in tests
Files:
tests/e2e/features/steps/info.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: build-pr
- GitHub Check: e2e_tests (ci)
- GitHub Check: e2e_tests (azure)
🔇 Additional comments (1)
tests/e2e/features/steps/info.py (1)
3-3: LGTM!The
jsonimport is appropriately added to support parsing expected tool structures from context text in the new step definition.
| assert ( | ||
| provider_tool["identifier"] == expected_json["identifier"] | ||
| ), f"identifier should be {expected_json["identifier"]}, but was {provider_tool["identifier"]}" | ||
| assert ( | ||
| provider_tool["description"] == expected_json["description"] | ||
| ), f"description should be {expected_json["description"]}" | ||
| assert ( | ||
| provider_tool["provider_id"] == expected_json["provider_id"] | ||
| ), f"provider_id should be {expected_json["provider_id"]}" | ||
| assert ( | ||
| provider_tool["toolgroup_id"] == expected_json["toolgroup_id"] | ||
| ), f"toolgroup_id should be {expected_json["toolgroup_id"]}" | ||
| assert ( | ||
| provider_tool["server_source"] == expected_json["server_source"] | ||
| ), f"server_source should be {expected_json["server_source"]}" | ||
| assert ( | ||
| provider_tool["type"] == expected_json["type"] | ||
| ), f"type should be {expected_json["type"]}" |
There was a problem hiding this comment.
Fix syntax errors in assertion messages caused by nested double quotes.
All assertion error messages in this function contain nested double quotes within f-strings, which will cause SyntaxError at runtime. The inner dictionary access uses double quotes that conflict with the f-string delimiters.
Apply this diff to fix all assertion messages:
assert (
provider_tool["identifier"] == expected_json["identifier"]
- ), f"identifier should be {expected_json["identifier"]}, but was {provider_tool["identifier"]}"
+ ), f"identifier should be {expected_json['identifier']}, but was {provider_tool['identifier']}"
assert (
provider_tool["description"] == expected_json["description"]
- ), f"description should be {expected_json["description"]}"
+ ), f"description should be {expected_json['description']}"
assert (
provider_tool["provider_id"] == expected_json["provider_id"]
- ), f"provider_id should be {expected_json["provider_id"]}"
+ ), f"provider_id should be {expected_json['provider_id']}"
assert (
provider_tool["toolgroup_id"] == expected_json["toolgroup_id"]
- ), f"toolgroup_id should be {expected_json["toolgroup_id"]}"
+ ), f"toolgroup_id should be {expected_json['toolgroup_id']}"
assert (
provider_tool["server_source"] == expected_json["server_source"]
- ), f"server_source should be {expected_json["server_source"]}"
+ ), f"server_source should be {expected_json['server_source']}"
assert (
provider_tool["type"] == expected_json["type"]
- ), f"type should be {expected_json["type"]}"
+ ), f"type should be {expected_json['type']}"🤖 Prompt for AI Agents
In tests/e2e/features/steps/info.py around lines 144 to 161, the f-string
assertion messages use nested double quotes which causes a SyntaxError; change
the inner dictionary access quotes to single quotes (e.g.
expected_json['identifier']) or escape the inner double quotes so each f-string
is valid, and update all assertion messages in this block consistently.
2 participants
Description
add new e2e tests for tools endpoint
Type of change
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit