[RFC]: Blackwell Enablement for vLLM (SM100)

[pavanimajety]

Motivation.

We are in the process of making incremental changes for Blackwell Support in vLLM. This issue is a tracker for all the items that are planned.

Planned or In Progress Features

The following items are either planned or currently in progress to enable vLLM support on Blackwell.

• Enable NVFP4 Support

  • (NVIDIA) Add functional support for NVFP4 Kernels for linear layers
  • (NVIDIA) Add functional support for NVFP4 MoE Kernels
  • (NVIDIA) Add Model Integration for nvidia/*-FP4 models
  • Finetune GEMM configurations for Blackwell
  • (NVIDIA) Optimize MoE for Latency
  • (NVIDIA) Optimize MoE for Throughput FI: PR !1113
  • (NVIDIA) MoE All Reduce Fusion FI: PR !1108

• Optimize communication overlap ops

  • (NVIDIA) Enable NCCL’s symmetric memory [core] add nccl symmetric memory for all reduce #24532
  • (NVIDIA) Add support for Gemm + comm overlap

• Blackwell Attention Kernels

  • (NVIDIA) Integrate Cutlass MLA Kernels [NVIDIA] Add Cutlass MLA backend #17625
  • (NVIDIA) Integrate vLLM v1-compatible Blackwell prefill and decode GQA kernels FI: PR !1051

• FP8 Blockscale Gemm and MoE

  • (NVIDIA) FP8 Blockscale GEMM
  • (NVIDIA) FP8 Blockscale gemm optimizations: Sm100 blockwise fp8 swap ab #18564
  • (NVIDIA) FP8 Blockscale MoE
  • (NVIDIA) Latency and throughput optimizations

• MTP support

Feedback Period.

No response

CC List.

@kushanam @kaixih

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tlrmchlsmth]

Any more details on your plans for NCCL Symmetric memory + GEMM-Comms overlap?

Check out this PR which is doing work along those lines #17882

[kushanam]

Any more details on your plans for NCCL Symmetric memory

the symmetric memory will be for low latency communications. Please stay tuned!

GEMM-Comms overlap?

We are working on enabling GEMM+AR overlap as well as MoE with in-kernel comms again for low latency cases.

[cyril23]

thanks a lot for your support for the community!

[x] FP8 Blockscale GEMM

Is that one done already? May I ask how to use FP8 with Blackwell currently? What I've tried so far:

git clone https://github.com/vllm-project/vllm
cd vllm/
git checkout tags/v0.9.0 -b mybranch
sudo docker build . --target vllm-openai \
  --tag myvllm --file docker/Dockerfile \
  --build-arg max_jobs=4 \
  --build-arg nvcc_threads=1 \
  --build-arg torch_cuda_arch_list="12.0 12.1" \
  --build-arg RUN_WHEEL_CHECK=false

sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN" \
    myvllm-fp8 --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8
# error: EngineCore failed to start.

# Same error with another model too:
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN" \
    myvllm --model nvidia/Llama-3.1-8B-Instruct-FP8 \
	--max-model-len 8k \
	--quantization modelopt
# error: EngineCore failed to start.

Error details and system info: 5090-vllm090-fp8-errordetails.txt

[pavanimajety]

Is that one done already? May I ask how to use FP8 with Blackwell currently? What I've tried so far:

@cyril23 Hey Niko, this issue is for SM100 - B200 series of GPUs. SM120(5080, 5090) Blackwell support still needs to be added for GEMMS to vLLM. But on the bright side CUTLASS examples and support has released for 80_blackwell_geforce_sparse_gemm and 79_blackwell_geforce_gemm. We would love some community support here.

[mratsim]

The get_min_capability code should be refactored to attributes like has_xxx_extension

vllm/vllm/model_executor/layers/quantization/kernels/mixed_precision/machete.py

Lines 20 to 24 in 8f8900c

	class MacheteLinearKernel(MPLinearKernel):
	@classmethod
	def get_min_capability(cls) -> int:
	return 90

The min-capabilities detection will lump SM120 (consumer and workstation Blackwell) with SM90a (Hopper) but both SM90a and SM100a have dedicated instructions.

See MacheteLinearKernel in #16131

[houseroad]

cc: @chenyang78, @drisspg for the Blackwell enablement.

[bely66]

Hi @pavanimajety

Any idea if the roadmap will cover issues with enabling lora on RTX-5090 GPUs because it seems not to be working at all and it can't recognize the kernels

[XiaoSong9905]

Any more details on your plans for NCCL Symmetric memory

the symmetric memory will be for low latency communications. Please stay tuned!

GEMM-Comms overlap?

We are working on enabling GEMM+AR overlap as well as MoE with in-kernel comms again for low latency cases.

Hi @kushanam , thats for the Blackwell integration! Can you help provide more detail regarding We are working on enabling GEMM+AR overlap ?

• Does GEMM and AR here have data dependency (GEMM's output is AR's input) ?
• Are u planning to Fuse GEMM & AR into single kernel?
• Are u planning to use multi-stream approach to launch multiple GEMM and multiple AR?