Skip to content

Fix timeout error in the FastAPI frontend #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 19, 2023
+5 −2
Merged

Fix timeout error in the FastAPI frontend #34

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c61ee87
Fix timeout error in the FastAPI frontend
zhuohan123 Apr 9, 2023
017d780
modify pass to continue
zhuohan123 Apr 9, 2023
b971503
Merge branch 'main' into fix-timeout-error
zhuohan123 May 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions cacheflow/frontend/fastapi_frontend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ async def server_step(self):
self.is_server_running = True
updated_seq_groups = await self.server.step.remote()
self.is_server_running = False
# Notify the waiting coroutines that there new outputs ready.
# Notify the waiting coroutines that there are new outputs ready.
for seq_group in updated_seq_groups:
group_id = seq_group.group_id
self.running_seq_groups[group_id] = seq_group
Expand Down Expand Up @@ -121,7 +121,10 @@ async def generate(self, request_dict: Dict):
# Wait for new output. The group_event will be set in server_step
# when there is new output available for the sequence group.
# Added a timeout to prevent deadlock.
await asyncio.wait_for(group_event.wait(), timeout=TIMEOUT_TO_PREVENT_DEADLOCK)
try:
await asyncio.wait_for(group_event.wait(), timeout=TIMEOUT_TO_PREVENT_DEADLOCK)
except asyncio.TimeoutError:
continue
# Reset the event to wait for the next output.
group_event.clear()
# Decode and return new outputs
Expand Down