[Doc] Update docs on handling OOM

[DarkLight1337]

• Specifically ask users to set --max-num-seqs to avoid OOM in V1
• Bring up more options (VLLM_MM_INPUT_CACHE_GIB and VLLM_CPU_KVCACHE_SPACE) to reduce CPU memory consumption.
• Reduce VLLM_MM_INPUT_CACHE_GIB default to 4 (previous 8) as users with 32GB RAM may otherwise run out of memory.
• Misc: Update Engine Arguments page to point back to offline inference and online serving pages for easy reference.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

cc @robertgshaw2-redhat should we reduce the default max_num_seqs for V1, or adjust it automatically somehow? There have been numerous reports of OOM from people using lower-end GPUs like RTX3060.

[robertgshaw2-redhat]

cc @robertgshaw2-redhat should we reduce the default max_num_seqs for V1, or adjust it automatically somehow? There have been numerous reports of OOM from people using lower-end GPUs like RTX3060.

Why does --max-num-seqs result in OOM?

[ywang96]

cc @robertgshaw2-redhat should we reduce the default max_num_seqs for V1, or adjust it automatically somehow? There have been numerous reports of OOM from people using lower-end GPUs like RTX3060.

Why does --max-num-seqs result in OOM?

@robertgshaw2-redhat This is mostly related to two changes we made on V1

1. max-num-seqs was raised from 256 to 1024 by default
2. On V1 there were two PRs added late in the development ([Bugfix] V1 Memory Profiling: V0 Sampler Integration without Rejection Sampler #13594, [V1][Core] Fix memory issue with logits & sampling #14508) to add sampler dummy run into profile_run and compile_or_warm_up_model (the latter is to mitigate memory fragmentation issue which this was previously missing on V0). This means there will always be a dummy sampler run with max number of possible decoding sequences (which's very often --max-num-seqs), and can sometimes result in OOM becasue of the new default. We have also gave an explicit error about it here but users can sometimes miss it.

   vllm/vllm/v1/worker/gpu_model_runner.py

   Lines 1353 to 1357 in dccf535

   	raise RuntimeError(
   	"CUDA out of memory occurred when warming up sampler with "
   	f"{num_reqs} dummy requests. Please try lowering "
   	"`max_num_seqs` or `gpu_memory_utilization` when "
   	"initializing the engine.") from e

[robertgshaw2-redhat]

cc @robertgshaw2-redhat should we reduce the default max_num_seqs for V1, or adjust it automatically somehow? There have been numerous reports of OOM from people using lower-end GPUs like RTX3060.

Why does --max-num-seqs result in OOM?

@robertgshaw2-redhat This is mostly related to two changes we made on V1

1. max-num-seqs was raised from 256 to 1024 by default
2. On V1 there were two PRs added late in the development ([Bugfix] V1 Memory Profiling: V0 Sampler Integration without Rejection Sampler #13594, [V1][Core] Fix memory issue with logits & sampling #14508) to add sampler dummy run into profile_run and compile_or_warm_up_model (the latter is to mitigate memory fragmentation issue which this was previously missing on V0). This means there will always be a dummy sampler run with max number of possible decoding sequences (which's very often --max-num-seqs), and can sometimes result in OOM becasue of the new default. We have also gave an explicit error about it here but users can sometimes miss it.

   vllm/vllm/v1/worker/gpu_model_runner.py

   Lines 1353 to 1357 in dccf535

   	raise RuntimeError(
   	"CUDA out of memory occurred when warming up sampler with "
   	f"{num_reqs} dummy requests. Please try lowering "
   	"`max_num_seqs` or `gpu_memory_utilization` when "
   	"initializing the engine.") from e

Okay, we can set --max-num-seqs 1024 just for H100 and A100 then. WDYT @WoosukKwon

[houseroad]

Feel it's safer to leave the default unchanged if it's already used in production.

[ywang96]

Suggested change

	The default `max_num_seqs` has been raised from `256` in V0 to `1024` in V1.
	If you encounter OOM only when using V1 engine, try setting a lower value of `max_num_seqs`.
	The default `max_num_seqs` has been raised from `256` in V0 to `1024` in V1.
	If you encounter CUDA OOM only when using V1 engine, try setting a lower value of `max_num_seqs` or `gpu_memory_utilization`.

[DarkLight1337]

I am worried this might lead to some confusion as lowering gpu_memory_utilization may lead to a related error "No available memory for the cache blocks"

[ywang96]

Let’s still clarify this is related to CUDA OOM

[DarkLight1337]

Updated in 4c27e09

[ywang96]

@simon-mo What's your thought on this one?

[ywang96]

Now I think about it - maybe we move this to V1 User Guide page?

[ywang96]

I moved this under FAQ for V1

[GohioAC]

The OOM issue is not limited to the dummy sampler run with max number of possible decoding sequences.

I'm running some 1B multimodal models with 32 max-num-seqs, and the CPU RAM usage increases after every batch. I have tried deleting previous batch inputs and garbage collecting to no avail. Even setting disable_mm_preprocessor_cache to True does not help.

Let me know, if I should create a dedicated issue with more details. This issue is specific to the V1 engine.

[DarkLight1337]

Which version of vLLM are you using? Both v0.8.1 and v0.8.2 fixed some memory leaks.

[GohioAC]

Just saw #15294. Looks like the same issue.
My bad for not searching thoroughly. I'll give v0.8.2 a whirl.