[Bugfix] Temporary fix for quantization + CPU offloading

[heheda12345]

#17945 breaks the quantization + CPU offloading test. The reason is that #17945 delays the initialization of GPUModelRunner.InputBatch from GPUModelRunner.__init__ to GPUModelRunner.initialize_kv_cache and triggers some unknown bug in quantization + CPU offloading. This PR provides a temporary fix by moving the initialization of some tensors back to GPUModelRunner.__init__.

I believe the CI failure is not caused by a bug in #17945 because #18298 that only moves the input batch fails on the same test.

pls revert this PR after finding the root cause.

Related: #18425, #18459

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[heheda12345]

CC @DarkLight1337 @markmc @WoosukKwon

[heheda12345]

Add ready to trigger full CI

[DarkLight1337]

Approving to unblock CI

[DarkLight1337]

This PR is now causing V1 test to fail

[DarkLight1337]

We are pausing PRs for 24 hrs, will this be enough time to figure out the root cause?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @heheda12345.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[heheda12345]

Just fixed the v1 test. Let's try again.
Just note that the behavior of quantization test failure was changed. Not sure whether there are more bugs in main branch now.
In #18425

FAILED quantization/test_cpu_offload.py::test_cpu_offload_gptq - RuntimeError: Server exited unexpectedly.
FAILED quantization/test_cpu_offload.py::test_cpu_offload_awq - RuntimeError: Server exited unexpectedly.
FAILED quantization/test_cpu_offload.py::test_cpu_offload_compressed_tensors - AssertionError: Results for model='nm-testing/llama7b-one-shot-2_4-w4a16-marlin24-t' are not the same.

In e60f550 (the main branch commit of #17945)

[2025-05-16T16:32:49Z] FAILED quantization/test_cpu_offload.py::test_cpu_offload_gptq - AssertionError: Results for model='Qwen/Qwen2-1.5B-Instruct-GPTQ-Int4' are not the same.
[2025-05-16T16:32:49Z] ref_args=['--quantization', 'gptq'] ref_envs=None
[2025-05-16T16:32:49Z] compare_args=['--quantization', 'gptq', '--cpu-offload-gb', '1'] compare_envs=None
[2025-05-16T16:32:49Z] ref_result={'test': 'seeded_sampling', 'text': " Mike and I'm a", 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=5, total_tokens=10, completion_tokens_details=None, prompt_tokens_details=None)}
[2025-05-16T16:32:49Z] compare_result={'test': 'seeded_sampling', 'text': ' Sarah and this will be', 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=5, total_tokens=10, completion_tokens_details=None, prompt_tokens_details=None)}
[2025-05-16T16:32:49Z] FAILED quantization/test_cpu_offload.py::test_cpu_offload_awq - AssertionError: Results for model='Qwen/Qwen2-1.5B-Instruct-AWQ' are not the same.
[2025-05-16T16:32:49Z] ref_args=['--quantization', 'awq'] ref_envs=None
[2025-05-16T16:32:49Z] compare_args=['--quantization', 'awq', '--cpu-offload-gb', '1'] compare_envs=None
[2025-05-16T16:32:49Z] ref_result={'test': 'single_completion', 'text': ' John and I am a', 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=5, total_tokens=10, completion_tokens_details=None, prompt_tokens_details=None)}
[2025-05-16T16:32:49Z] compare_result={'test': 'single_completion', 'text': ' Kaitlyn and I', 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=5, total_tokens=10, completion_tokens_details=None, prompt_tokens_details=None)}
[2025-05-16T16:32:49Z] FAILED quantization/test_cpu_offload.py::test_cpu_offload_compressed_tensors - AssertionError: Results for model='nm-testing/llama7b-one-shot-2_4-w4a16-marlin24-t' are not the same.
[2025-05-16T16:32:49Z] ref_args=[] ref_envs=None
[2025-05-16T16:32:49Z] compare_args=['--cpu-offload-gb', '1'] compare_envs=None
[2025-05-16T16:32:49Z] ref_result={'test': 'single_completion', 'text': ' ... ... . Today I', 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=6, total_tokens=11, completion_tokens_details=None, prompt_tokens_details=None)}
[2025-05-16T16:32:49Z] compare_result={'test': 'single_completion', 'text': ' ... ... .\n I', 'finish_reason': 'length', 'usage': CompletionUsage(completion_tokens=5, prompt_tokens=6, total_tokens=11, completion_tokens_details=None, prompt_tokens_details=None)}

Here:

[2025-05-21T16:45:31Z] FAILED quantization/test_bitsandbytes.py::test_load_8bit_bnb_model[meta-llama/Llama-Guard-3-8B-INT8-read pre-quantized llama 8-bit model] - AssertionError: function <function test_load_8bit_bnb_model at 0x7f61d5649800> failed when called with args () and kwargs {'hf_runner': <class 'tests.conftest.HfRunner'>, 'vllm_runner': <class 'tests.conftest.VllmRunner'>, 'example_prompts': ['vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n', 'Briefly describe the major milestones in the development of artificial intelligence from 1950 to 2020.\n', 'Compare and contrast artificial intelligence with human intelligence in terms of processing information.\n', 'Describe the basic components of a neural network and how it can be trained.\n', 'Write a short story about a robot that dreams for the first time.\n', 'Analyze the impact of the COVID-19 pandemic on global economic structures and future business models.\n', 'Explain the cultural significance of the Mona Lisa painting, and how its perception might vary in Western versus Eastern societies.\n', "Translate the following English sentence into Japanese, French, and Swahili: 'The early bird catches the worm.'\n"], 'model_name': 'meta-llama/Llama-Guard-3-8B-INT8', 'description': 'read pre-quantized llama 8-bit model'}
[2025-05-21T16:45:31Z] FAILED quantization/test_cpu_offload.py::test_cpu_offload_gptq - RuntimeError: Server exited unexpectedly.
[2025-05-21T16:45:31Z] FAILED quantization/test_cpu_offload.py::test_cpu_offload_awq - RuntimeError: Server exited unexpectedly.
[2025-05-21T16:45:31Z] FAILED quantization/test_fp8.py::test_model_load_and_run[False-True-neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV] - vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

[heheda12345]

close this PR as we choose to revert #17945