Skip to content

Conversation

@youkaichao
Copy link
Member

fixes #6194

quite a lot users don't know the difference between simple api server and openai api server. I got several issue reports recently, that users are manually getting /generate from openai api server, or getting /v1/models from simple api server, and then complain about 404 response.

logging all available endpoints should help users clearly know what endpoints they can test.

example output:

simple api server: python -m vllm.entrypoints.api_server

INFO 07-07 13:50:36 api_server.py:116] Available endpoints are:
INFO 07-07 13:50:36 api_server.py:120] Path: /openapi.json, Methods: GET, HEAD
INFO 07-07 13:50:36 api_server.py:120] Path: /docs, Methods: GET, HEAD
INFO 07-07 13:50:36 api_server.py:120] Path: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-07 13:50:36 api_server.py:120] Path: /redoc, Methods: GET, HEAD
INFO 07-07 13:50:36 api_server.py:120] Path: /health, Methods: GET
INFO 07-07 13:50:36 api_server.py:120] Path: /generate, Methods: POST

openai api server: python -m vllm.entrypoints.openai.api_server

INFO 07-07 13:53:48 api_server.py:244] Available endpoints are:
INFO 07-07 13:53:48 api_server.py:248] Path: /openapi.json, Methods: HEAD, GET
INFO 07-07 13:53:48 api_server.py:248] Path: /docs, Methods: HEAD, GET
INFO 07-07 13:53:48 api_server.py:248] Path: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-07 13:53:48 api_server.py:248] Path: /redoc, Methods: HEAD, GET
INFO 07-07 13:53:48 api_server.py:248] Path: /metrics, Methods: No methods
INFO 07-07 13:53:48 api_server.py:248] Path: /health, Methods: GET
INFO 07-07 13:53:48 api_server.py:248] Path: /tokenize, Methods: POST
INFO 07-07 13:53:48 api_server.py:248] Path: /detokenize, Methods: POST
INFO 07-07 13:53:48 api_server.py:248] Path: /v1/models, Methods: GET
INFO 07-07 13:53:48 api_server.py:248] Path: /version, Methods: GET
INFO 07-07 13:53:48 api_server.py:248] Path: /v1/chat/completions, Methods: POST
INFO 07-07 13:53:48 api_server.py:248] Path: /v1/completions, Methods: POST
INFO 07-07 13:53:48 api_server.py:248] Path: /v1/embeddings, Methods: POST

@youkaichao youkaichao requested a review from comaniac July 7, 2024 21:31
Copy link
Collaborator

@comaniac comaniac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@youkaichao
Copy link
Member Author

merge as failed tests are caused by hf connection error.

@youkaichao youkaichao merged commit 3b08fe2 into vllm-project:main Jul 7, 2024
@youkaichao youkaichao deleted the log_endpoints branch July 7, 2024 22:11
xjpang pushed a commit to xjpang/vllm that referenced this pull request Jul 24, 2024
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: No end point available after model is fully loaded

2 participants