Llama3.2 Vision Model: Guides and Issues

[simon-mo]

Running the server (using the vLLM CLI or our docker image):

• vllm serve meta-llama/Llama-3.2-11B-Vision-Instruct --enforce-eager --max-num-seqs 16
• vllm serve meta-llama/Llama-3.2-90B-Vision-Instruct --enforce-eager --max-num-seqs 32 --tensor-parallel-size 8

Currently:

• Only one leading image is supported. Support for multiple images and interleaving images are work in progress.
• Text only inference is supported.
• Only NVIDIA GPUs are supported.
• Performance is acceptable but to be optimized! We aim at first release to be functionality correct. We will work on making it fast 🏎️

Please see the next steps for better supporting this model on vLLM.

cc @heheda12345 @ywang96

[shermansiu]

What is the ETA for adding support for multiple images? For a project I'm working on, we're trying to see if it's viable to use vLLM for Llama 3.2 in the next few days.

[pseudotensor]

What is optimal way to use H100s? I'd guess 8 are not required.

I can get 4*H100's to start up like this:

docker pull vllm/vllm-openai:v0.6.2
docker stop llama31-70b ; docker remove llama31-70b
docker run -d --restart=always \
    --runtime=nvidia \
    --gpus '"device=1,2,3,7"' \
    --shm-size=10.24gb \
    -p 5020:5020 \
        -e NCCL_IGNORE_DISABLED_P2P=1 \
    -e HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN \
    -e VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2 \
    -v /etc/passwd:/etc/passwd:ro \
    -v /etc/group:/etc/group:ro \
    -u `id -u`:`id -g` \
    -v "${HOME}"/.cache:$HOME/.cache/ -v "${HOME}"/.config:$HOME/.config/   -v "${HOME}"/.triton:$HOME/.triton/  \
    --network host \
    --name llama31-70b \
     vllm/vllm-openai:v0.6.2 \
        --port=5020 \
        --host=0.0.0.0 \
        --model=meta-llama/Llama-3.2-90B-Vision-Instruct \
        --seed 1234 \
        --tensor-parallel-size=4 \
        --max-model-len=81920 \
        --max-num-batched-tokens=81920 --max-log-len=100 \
        --limit_mm_per_prompt 'image=1' \
        --enforce-eager --max-num-seqs 8 \
        --gpu-memory-utilization 0.99 \
        --served-model-name meta-llama/Llama-3.2-90B-Vision-Instruct meta-llama/Meta-Llama-3.1-70B-Instruct \
        --download-dir=$HOME/.cache/huggingface/hub &>> logs.vllm_server.llama31_70b.txt

But it fails when used like this. Just for a test TEXT only query, no images. Seems very short queries work, but around 10k input fails.

WARNING 09-25 18:52:22 preprocess.py:86] Falling back on <BOS> for decoder start token id because decoder start token id is not available.
INFO 09-25 18:52:22 engine.py:288] Added request cmpl-7f02d431762140e784491182a7121772-0.
ERROR 09-25 18:52:23 engine.py:157] AssertionError()
ERROR 09-25 18:52:23 engine.py:157] Traceback (most recent call last):
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 155, in start
ERROR 09-25 18:52:23 engine.py:157]     self.run_engine_loop()
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 218, in run_engine_loop
ERROR 09-25 18:52:23 engine.py:157]     request_outputs = self.engine_step()
ERROR 09-25 18:52:23 engine.py:157]                       ^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 236, in engine_step
ERROR 09-25 18:52:23 engine.py:157]     raise e
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 227, in engine_step
ERROR 09-25 18:52:23 engine.py:157]     return self.engine.step()
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 1264, in step
ERROR 09-25 18:52:23 engine.py:157]     outputs = self.model_executor.execute_model(
ERROR 09-25 18:52:23 engine.py:157]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/distributed_gpu_executor.py", line 78, in execute_model
ERROR 09-25 18:52:23 engine.py:157]     driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 09-25 18:52:23 engine.py:157]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 148, in _driver_execute_model
ERROR 09-25 18:52:23 engine.py:157]     return self.driver_worker.execute_model(execute_model_req)
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 327, in execute_model
ERROR 09-25 18:52:23 engine.py:157]     output = self.model_runner.execute_model(
ERROR 09-25 18:52:23 engine.py:157]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 09-25 18:52:23 engine.py:157]     return func(*args, **kwargs)
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/enc_dec_model_runner.py", line 223, in execute_model
ERROR 09-25 18:52:23 engine.py:157]     output: SamplerOutput = self.model.sample(
ERROR 09-25 18:52:23 engine.py:157]                             ^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mllama.py", line 940, in sample
ERROR 09-25 18:52:23 engine.py:157]     next_tokens = self.sampler(logits, sampling_metadata)
ERROR 09-25 18:52:23 engine.py:157]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 09-25 18:52:23 engine.py:157]     return self._call_impl(*args, **kwargs)
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 09-25 18:52:23 engine.py:157]     return forward_call(*args, **kwargs)
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 274, in forward
ERROR 09-25 18:52:23 engine.py:157]     maybe_deferred_sample_results, maybe_sampled_tokens_tensor = _sample(
ERROR 09-25 18:52:23 engine.py:157]                                                                  ^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 879, in _sample
ERROR 09-25 18:52:23 engine.py:157]     return _sample_with_torch(
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 819, in _sample_with_torch
ERROR 09-25 18:52:23 engine.py:157]     multinomial_samples[sampling_type] = _multinomial(
ERROR 09-25 18:52:23 engine.py:157]                                          ^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 627, in _multinomial
ERROR 09-25 18:52:23 engine.py:157]     assert seq_group.generator is not None
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157] AssertionError
(VllmWorkerProcess pid=209) INFO 09-25 18:52:23 multiproc_worker_utils.py:244] Worker exiting
(VllmWorkerProcess pid=211) INFO 09-25 18:52:23 multiproc_worker_utils.py:244] Worker exiting
(VllmWorkerProcess pid=210) INFO 09-25 18:52:23 multiproc_worker_utils.py:244] Worker exiting
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 257, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 253, in wrap
    await func()
  File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 230, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
    await self.message_event.wait()
  File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7ff530597740

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/usr/local/lib/python3.12/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.12/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 250, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.12/dist-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 253, in wrap
    |     await func()
    |   File "/usr/local/lib/python3.12/dist-packages/starlette/responses.py", line 242, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 248, in completion_stream_generator
    |     async for prompt_idx, res in result_generator:
    |   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 490, in merge_async_iterators
    |     item = await d
    |            ^^^^^^^
    |   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/client.py", line 486, in _process_request
    |     raise request_output
    | AssertionError
    +------------------------------------
INFO 09-25 18:52:24 multiproc_worker_utils.py:124] Killing local vLLM worker processes
[rank0]:[W925 18:52:26.595501679 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
INFO:     172.16.0.87:35352 - "GET /health HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application

[tcapelle]

How can I run the vllm docker image against the meta provided checkpoint (not the huggingface one)?

[groccy]

What is the ETA for adding support for multiple images? For a project I'm working on, we're trying to see if it's viable to use vLLM for Llama 3.2 in the next few days.

Same here. If we can know the ETA for adding support for multiple images and interleaving images, it will be truly appreciated. We also have a project that depends on this kind of SOTA VLMs.

Thank you so much for your hard work and great contributions!

[heheda12345]

How can I run the vllm docker image against the meta provided checkpoint (not the huggingface one)?

You can add the logic of loading meta provided checkpoint here. The main work will be changing some names of the parameter. And you still need huggingface's configs.

[tcapelle]

I got a HF checkpoint thatnks to some EU friends, managed to serve the model half of today, but now it's dying

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb9eaf03f86 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb9eaeb2d10 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb9eafdef08 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb9ec1fb3e6 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fb9ec200600 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fb9ec2072ba in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb9ec2096fc in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xd6df4 (0x7fba399addf4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7fba3abc6609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7fba3ad00353 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG 3 Rank 0] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb9eaf03f86 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb9eaeb2d10 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb9eafdef08 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb9ec1fb3e6 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fb9ec200600 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fb9ec2072ba in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb9ec2096fc in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xd6df4 (0x7fba399addf4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x8609 (0x7fba3abc6609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7fba3ad00353 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1521 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb9eaf03f86 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe5aa84 (0x7fb9ebe92a84 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xd6df4 (0x7fba399addf4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x8609 (0x7fba3abc6609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x43 (0x7fba3ad00353 in /usr/lib/x86_64-linux-gnu/libc.so.6)

INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown

[sfc-gh-zhwang]

What is the general memory requirement for the model. Seems that 90b model is not able to run on an 8x40GB A100 machine.

[varoudis]

What is the general memory requirement for the model. Seems that 90b model is not able to run on an 8x40GB A100 machine.

pif... imagine I tried to run it on 8 A30 :(

[DarkLight1337]

If you run into OOM issues, please reduce max_model_len and/or max_num_seqs, as described in the example script.

[ywang96]

Thank you all for trying out Llama 3.2 vision model on vLLM!

As you may already know, multimodal Llama 3.2 is quite different from other LlaVA-style VLMs that we currently support on vLLM as it involves cross-attention layers in the language model, and below are the next steps and their priorities for features and optimizations related to this new architecture.

P0

• Cross attention mask:
  • To properly support multi/interleaved image inference, a custom cross attention mask is required but passing attention mask to attention module is not supported on vLLM today. Help wanted!
  • [Model] Make llama3.2 support multiple and interleaved images #9095
• Address KV cache waste issue:
  • We need better memory management for cross attention/encoder-decoder models since the page size cannot be specified separately.
• Optimize inference of mixed requests (text-only & text + image):
  • Since the model forward pass changes dynamically to the input (text only or text + image), we needed to add additional dummy image + mask for text-only sequences so that mixed requests can be batched
  • As a result, there's significant room for optimizing both compute & space efficiency.
  • This is less of an issue for offline inference (where typically requests are either all text-only or all text + image), but more for the online serving case.

P1

• Attention backends other than xformers for cross attention
  • [Encoder Decoder] Add flash_attn kernel support for encoder-decoder models #9559
  • [Encoder Decoder] Update Mllama to run with both FlashAttention and XFormers #9982
• Enable CUDA graph support for llama 3.2 vision #14917

P2

• Image embeddings as input
• Frontend changes for multi/interleave images
  • [Frontend] Enable Online Multi-image Support for MLlama #9393
• Add xformers implementation for vision encoder attention

[pseudotensor]

Any thoughts on my issue or should I post real separate issue?

ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 819, in _sample_with_torch
ERROR 09-25 18:52:23 engine.py:157]     multinomial_samples[sampling_type] = _multinomial(
ERROR 09-25 18:52:23 engine.py:157]                                          ^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sampler.py", line 627, in _multinomial
ERROR 09-25 18:52:23 engine.py:157]     assert seq_group.generator is not None
ERROR 09-25 18:52:23 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 09-25 18:52:23 engine.py:157] AssertionError

#8826 (comment)