Skip to content

Update Envoy config with longer timeouts #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
liu-cong opened this issue Oct 21, 2024 · 3 comments · Fixed by #39
Closed

Update Envoy config with longer timeouts #24

liu-cong opened this issue Oct 21, 2024 · 3 comments · Fixed by #39

Comments

@liu-cong
Copy link
Contributor

liu-cong commented Oct 21, 2024 •
edited
Loading

During benchmarking I noticed as we increase QPS the request start to take longer, and I started to notice 500 errors from Envoy. @kaushikmitr had some timeout config in the poc that reduces such errors significantly. This task is to update the Envoy config with proper timeout config and document them.

### Tasks
@liu-cong
Copy link
Contributor Author

@kaushikmitr maybe you can take this?

@kaushikmitr
Copy link
Contributor

sure I can take this up

@kfswain kfswain mentioned this issue Nov 18, 2024
Merged
@liu-cong
Copy link
Contributor Author

closed by #39

shaneutt added a commit to shaneutt/gateway-api-inference-extension that referenced this issue Apr 18, 2025
@shaneutt
Merge pull request kubernetes-sigs#24 from mayabar/dev
8004347 
Add inference model and pool yamls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adding circuit breaker and timeout layers to avoid Gateway 5xx errors. kfswain/llm-instance-gateway
2 participants
@liu-cong @kaushikmitr