Skip to content

Conversation

@leseb
Copy link
Collaborator

@leseb leseb commented Nov 19, 2025

What does this PR do?

This commit introduces a new FastAPI router-based system for defining API endpoints, enabling a migration path away from the legacy @webmethod decorator system. The implementation includes router infrastructure, migration of the Batches API as the first example, and updates to server, OpenAPI generation, and inspection systems to support both routing approaches.

The router infrastructure consists of a router registry system that allows APIs to register FastAPI router factories, which are then automatically discovered and included in the server application. Standard error responses are centralized in router_utils to ensure consistent OpenAPI specification generation with proper $ref references to component responses.

The Batches API has been migrated to demonstrate the new pattern. The protocol definition and models remain in llama_stack_api/batches, maintaining clear separation between API contracts and server implementation. The FastAPI router implementation lives in
llama_stack/core/server/routers/batches, following the established pattern where API contracts are defined in llama_stack_api and server routing logic lives in
llama_stack/core/server.

The server now checks for registered routers before falling back to the legacy webmethod-based route discovery, ensuring backward compatibility during the migration period. The OpenAPI generator has been updated to handle both router-based and webmethod-based routes, correctly extracting metadata from FastAPI route decorators and Pydantic Field descriptions. The inspect endpoint now includes routes from both systems, with proper filtering for deprecated routes and API levels.

Response descriptions are now explicitly defined in router decorators, ensuring the generated OpenAPI specification matches the previous format. Error responses use $ref references to component responses (BadRequest400, TooManyRequests429, etc.) as required by the specification. This is neat and will allow us to remove a lot of boiler plate code from our generator once the migration is done.

This implementation provides a foundation for incrementally migrating other APIs to the router system while maintaining full backward compatibility with existing webmethod-based APIs.

Closes: #4188

Test Plan

CI, the server should start, same routes should be visible.

curl http://localhost:8321/v1/inspect/routes | jq '.data[] | select(.route | contains("batches"))'

Also:

 uv run pytest tests/integration/batches/ -vv --stack-config=http://localhost:8321
================================================== test session starts ==================================================
platform darwin -- Python 3.12.8, pytest-8.4.2, pluggy-1.6.0 -- /Users/leseb/Documents/AI/llama-stack/.venv/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.8', 'Platform': 'macOS-26.0.1-arm64-arm-64bit', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.9.0', 'html': '4.1.1', 'socket': '0.7.0', 'asyncio': '1.1.0', 'json-report': '1.5.0', 'timeout': '2.4.0', 'metadata': '3.1.1', 'cov': '6.2.1', 'nbval': '0.11.0'}}
rootdir: /Users/leseb/Documents/AI/llama-stack
configfile: pyproject.toml
plugins: anyio-4.9.0, html-4.1.1, socket-0.7.0, asyncio-1.1.0, json-report-1.5.0, timeout-2.4.0, metadata-3.1.1, cov-6.2.1, nbval-0.11.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 24 items                                                                                                      

tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_creation_and_retrieval[None] SKIPPED [  4%]
tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_listing[None] SKIPPED               [  8%]
tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_immediate_cancellation[None] SKIPPED [ 12%]
tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_e2e_chat_completions[None] SKIPPED  [ 16%]
tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_e2e_completions[None] SKIPPED       [ 20%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_invalid_endpoint[None] SKIPPED [ 25%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_cancel_completed[None] SKIPPED [ 29%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_missing_required_fields[None] SKIPPED [ 33%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_invalid_completion_window[None] SKIPPED [ 37%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_streaming_not_supported[None] SKIPPED [ 41%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_mixed_streaming_requests[None] SKIPPED [ 45%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_endpoint_mismatch[None] SKIPPED [ 50%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_missing_required_body_fields[None] SKIPPED [ 54%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_invalid_metadata_types[None] SKIPPED [ 58%]
tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_e2e_embeddings[None] SKIPPED        [ 62%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_nonexistent_file_id PASSED [ 66%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_malformed_jsonl PASSED     [ 70%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_file_malformed_batch_file[empty] XFAIL [ 75%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_file_malformed_batch_file[malformed] XFAIL [ 79%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_retrieve_nonexistent PASSED [ 83%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_cancel_nonexistent PASSED  [ 87%]
tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_error_handling_invalid_model PASSED [ 91%]
tests/integration/batches/test_batches_idempotency.py::TestBatchesIdempotencyIntegration::test_idempotent_batch_creation_successful PASSED [ 95%]
tests/integration/batches/test_batches_idempotency.py::TestBatchesIdempotencyIntegration::test_idempotency_conflict_with_different_params PASSED [100%]

================================================= slowest 10 durations ==================================================
1.01s call     tests/integration/batches/test_batches_idempotency.py::TestBatchesIdempotencyIntegration::test_idempotent_batch_creation_successful
0.21s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_nonexistent_file_id
0.17s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_malformed_jsonl
0.12s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_error_handling_invalid_model
0.05s setup    tests/integration/batches/test_batches.py::TestBatchesIntegration::test_batch_creation_and_retrieval[None]
0.02s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_file_malformed_batch_file[empty]
0.01s call     tests/integration/batches/test_batches_idempotency.py::TestBatchesIdempotencyIntegration::test_idempotency_conflict_with_different_params
0.01s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_file_malformed_batch_file[malformed]
0.01s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_retrieve_nonexistent
0.00s call     tests/integration/batches/test_batches_errors.py::TestBatchesErrorHandling::test_batch_cancel_nonexistent
======================================= 7 passed, 15 skipped, 2 xfailed in 1.78s ========================================

This commit introduces a new FastAPI router-based system for defining
API endpoints, enabling a migration path away from the legacy @webmethod
decorator system. The implementation includes router infrastructure,
migration of the Batches API as the first example, and updates
to server, OpenAPI generation, and inspection systems to
support both routing approaches.

The router infrastructure consists of a router registry system
that allows APIs to register FastAPI router factories, which
are then automatically discovered and included in the server
application. Standard error responses are centralized in
router_utils to ensure consistent OpenAPI specification
generation with proper $ref references to component responses.

The Batches API has been migrated to demonstrate the new
pattern. The protocol definition and models remain in
llama_stack_api/batches, maintaining clear separation between
API contracts and server implementation. The FastAPI router
implementation lives in
llama_stack/core/server/routers/batches, following the
established pattern where API contracts are defined in
llama_stack_api and server routing logic lives in
llama_stack/core/server.

The server now checks for registered routers before falling
back to the legacy webmethod-based route discovery, ensuring
backward compatibility during the migration period. The
OpenAPI generator has been updated to handle both router-based
and webmethod-based routes, correctly extracting metadata from
FastAPI route decorators and Pydantic Field descriptions. The
inspect endpoint now includes routes from both systems, with
proper filtering for deprecated routes and API levels.

Response descriptions are now explicitly defined in router decorators,
ensuring the generated OpenAPI specification matches the
previous format. Error responses use $ref references to
component responses (BadRequest400, TooManyRequests429, etc.)
as required by the specification. This is neat and will allow us to
remove a lot of boiler plate code from our generator once the
migration is done.

This implementation provides a foundation for incrementally migrating
other APIs to the router system while maintaining full backward
compatibility with existing webmethod-based APIs.

Closes: llamastack#4188
Signed-off-by: Sébastien Han <[email protected]>
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 19, 2025
@ashwinb
Copy link
Contributor

ashwinb commented Nov 19, 2025

Thanks for writing this in a way which provides for incremental migration!

This commit migrates the Batches API to use FastAPI routers directly in the
API package, removing the need for custom decorator systems and manual router
registration. The API package now defines FastAPI routers using standard
FastAPI route decorators, making it self-sufficient and eliminating dependencies
on the server package.

The router implementation has been moved from llama_stack/core/server/routers/batches.py
to llama_stack_api/batches/routes.py, where it belongs alongside the protocol
and models.

Standard error responses (standard_responses) have been moved from the server
package to llama_stack_api/router_utils.py, ensuring the API package can
define complete routers without server dependencies. FastAPI has been added
as an explicit dependency to the llama-stack-api package, making it an
intentional dependency rather than an implicit one.

Router discovery is now fully automatic. The server discovers routers by
checking for routes modules in each API package and looking for a create_router
function. This eliminates the need for manual registration and makes the system
scalable - new APIs with router modules are automatically discovered and used.

The router registry has been simplified to use automatic discovery instead of
maintaining a manual registry. The build_router function (renamed from
create_router to better reflect its purpose) discovers and combines router
factories with implementations to create the final router instances.

Exposing Routers from the API is nice for the Bring Your Own API use
case too.

Signed-off-by: Sébastien Han <[email protected]>
This commit refactors the Batches protocol to use Pydantic request
models for both create_batch and list_batches methods, improving
consistency, readability, and maintainability.

- create_batch now accepts a single CreateBatchRequest parameter instead
  of individual arguments. This aligns the protocol with FastAPI’s
  request model pattern, allowing the router to pass the request object
  directly without unpacking parameters. Provider implementations now
  access fields via request.input_file_id, request.endpoint, etc.

- list_batches now accepts a single ListBatchesRequest parameter,
  replacing individual query parameters. The model includes after and
  limit fields with proper OpenAPI descriptions. FastAPI automatically
  parses query parameters into the model for GET requests, keeping
  router code clean. Provider implementations access fields via
  request.after and request.limit.

Signed-off-by: Sébastien Han <[email protected]>
Signed-off-by: Sébastien Han <[email protected]>
Copy link
Collaborator

@cdoern cdoern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initial look, comment on the router+fastapi dependencies of llama_stack_api and a question about imports of these new packages.

Refactored the router to accept the implementation directly instead of
using the impl_getter pattern.
The caller already knows which API it's building a router for.for

Signed-off-by: Sébastien Han <[email protected]>
We already have an impl at this point, no need to validate this again.

Signed-off-by: Sébastien Han <[email protected]>
Signed-off-by: Sébastien Han <[email protected]>
All batch models are now exported from the top level for better
discoverability and IDE support.

Signed-off-by: Sébastien Han <[email protected]>
@leseb
Copy link
Collaborator Author

leseb commented Nov 21, 2025

@ashwinb @cdoern all comments have been addressed! PTAL

Copy link
Collaborator

@cdoern cdoern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one comment about a slightly more strict check for our API version but it is a nit. otherwise LGTM!

Replace dynamic import-based router discovery with an explicit hardcoded
list of APIs that have routers.

Signed-off-by: Sébastien Han <[email protected]>
Less indirection and clearer declarations.

Signed-off-by: Sébastien Han <[email protected]>
@dgellow
Copy link
Contributor

dgellow commented Nov 24, 2025

Not exactly sure why the stainless github action is failing, but we got alerted on it and will investigate

…eliminate duplication

Added create_query_dependency() and create_path_dependency() helpers
that automatically generate FastAPI dependency functions from Pydantic
models. This makes the models the single source of truth for field
types, descriptions, and defaults, eliminating duplication between
models.py and fastapi_routes.py.

Signed-off-by: Sébastien Han <[email protected]>
Signed-off-by: Sébastien Han <[email protected]>
@leseb leseb requested a review from ashwinb November 24, 2025 13:57
@github-actions
Copy link
Contributor

github-actions bot commented Nov 24, 2025

✱ Stainless preview builds

This PR will update the llama-stack-client SDKs with the following commit message.

feat: Implement FastAPI router system

Edit this comment to update it. It will appear in the SDK's changelogs.

llama-stack-client-node studio · code · diff

Your SDK built successfully.
generate ⚠️build ✅lint ✅test ✅

npm install https://pkg.stainless.com/s/llama-stack-client-node/095ff6de9766374ec3bbe049d094c2a6b72a8a6b/dist.tar.gz
⚠️ llama-stack-client-kotlin studio · code · diff

There was a regression in your SDK.
generate ⚠️lint ✅test ⏳

llama-stack-client-python studio · code · diff

generate ⚠️build ⏳lint ⏳test ⏳

llama-stack-client-go studio · code · diff

Your SDK built successfully.
generate ⚠️lint ❗test ❗

go get github.com/stainless-sdks/llama-stack-client-go@55039c72d0604e38f462d11872b99d9f1460cc7c

⏳ These are partial results; builds are still running.


This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
Last updated: 2025-11-24 14:09:31 UTC

@leseb
Copy link
Collaborator Author

leseb commented Nov 24, 2025

I think Stainless struggles when the PR is not up to date with main. Now it's happy/

Copy link
Collaborator

@cdoern cdoern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for cleaning it up and addressing reviews. seems that the tracing.py stuff was removed so that clears up my only comment really.

I really like the dependency function generation!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, this is pretty clean, I like this improvement!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, it is rather sweet!

@dgellow
Copy link
Contributor

dgellow commented Nov 24, 2025

I think Stainless struggles when the PR is not up to date with main. Now it's happy

Ah ok, I arrived to the same conclusion and was going to post about it. Thanks, we will need to handle the situation more gracefully

}


def build_router(api: "Api", impl: Any) -> APIRouter | None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: build_fastapi_router at least for now

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


from fastapi import APIRouter

if TYPE_CHECKING:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this? why not just have a clear dependency here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hum I don't remember, probably a leftover.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# Add new APIs here as they are migrated to the router system
from llama_stack_api.batches.fastapi_routes import create_router as create_batches_router

_ROUTER_FACTORIES: dict[str, Any] = {
Copy link
Contributor

@ashwinb ashwinb Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just dict[Api, Callable[Any, ApiRouter]]?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


from typing import Protocol, runtime_checkable

try:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this specialized error message here?! if we did this, we'd need to this for every dependency we need? :) I suspect your LLM needs some instructions about defensiveness.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this code was already there, I just moved it.

@@ -0,0 +1,75 @@
# Copyright (c) Meta Platforms, Inc. and affiliates.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a fan of keeping this in the init.py -- why can't we move it to batches/api.py or batches/batches.py?

secondly, why do we want to split fastapi_routes.py (i.e., the annotations) from the API signature? I guess it is cleaner but now there is boilerplate and we need to keep these two things in sync!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a tradeoff, request/response types are already shared only the signatures are duplicated. We still need the protocol to be correctly exported for impls. We could use fastapi as a source of truth and generate the protocol from it, the other way around seems much more complex. However, it's more common to use the protocol as the reference since it could be used to write other implementation than fastapi. However since we only do fastapi it might make sense to consider the routes as source of truth.
However, doing this introduces a generator which might grow in complexity over time 🤷🏻 . Also, we not only need to generate the batches.py file but edit the __init__.py too in order to add the import eg from llama_stack_api.batches.batches import Batches. I'm not really keen on editing the init also... And since it's an API should rely on __all__ and not do import *.


from llama_stack_api.schema_utils import json_schema_type

try:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing about imports

for route in router.routes:
# Extract HTTP methods from the route
# FastAPI routes have methods as a set
if hasattr(route, "methods") and route.methods:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this code block so complex and has lots of FastAPI thingies when this branch is not even taken for FastAPI routers?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need this to expose the routes when getting http://localhost:8321/v1/inspect/routes, i'll simply the code

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return not route_deprecated

# Process router-based routes
from llama_stack.core.resolver import api_protocol_map
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the import inline?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no good reason, will move up

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

router = build_router(api, impl)
if router:
app.include_router(router)
logger.debug(f"Registered router for {api} API")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe say "Registered FastAPI router"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@ashwinb ashwinb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting there...

@leseb leseb requested a review from ashwinb November 25, 2025 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Foundation work using the batches API as example

4 participants