[CB] clean up request metadata for last submitted request

[yannicks1]

clean up request metadata for last submitted request

I have noticed that the request metadata, namely self.req_ids2blocks, self.req_ids2left_pads and self.free_blocks are not freed up/ updated for the last submitted request. This should be solved to avoid running out of blocks when using continuous batching in server mode.

Note: code currently on branch dev-updated-fms-api

[yannicks1]

I believe a possible solution to the above could also solve this in the same go.

[wallashss]

BTW:

I believe a possible solution to the above could also solve #82 (comment) in the same go.

I changed that in #127, I went back to your previous solution which is also what vllm does. vLLM has a case where you can call execute model with an "empty" scheduler output with just the request ids to finish. In this case we can properly clean up the data in the model runner side.

[wallashss]

I did some testing, and I got some surprises:

• The online serving is doing fine. All data is cleaned when all requests finish.
• The offline does not. I debugged and checked that there are still request metadata (self.req_ids2blocks, self.req_ids2left_pads and self.free_blocks). What is more surprising is that vllm upstream also has this problem in offline mode. I guess it is the way how the engine work wrapped by the llm class. In a nutshell it keeps running steps until the scheduler has pending requests, but in the "last step" it only puts in the queue to be released in a next step that is never executed. The model runner does not have a chance to release the metadata from the last request.

It is not good to have pending data to clean, but I think it is not so bad since it only in offline mode. if one send calls another llm.generate it will release in the next step and everything should keeping working in a fresh state. I think the solution is work with upstream to report/fix the issue and then update the vllm dependency to get the fix.

[prashantgupta24]

cc @sducouedic this would be great to add a test for!

[wallashss]

UPDATE:

I talked with @nikolaospapandreou and we observed the issue of the missing cleaning with offline mode that I commented early only happens if we disable vllm multiprocessing with VLLM_ENABLE_V1_MULTIPROCESSING=0.

So I guess everything is fine. I suggest that we can close now this issue. Even though there's a potential issue without multiprocessing, I think it is an edge case that we should not worry about it for now.

[yannicks1]

okay, thanks for the investigation! Do you agree @nikolaospapandreou that we can close this issue?

[prashantgupta24]

Should we close this issue once we have a good test case for it that proves it works?