SDPA re-compile during each SGLang forward

[airMeng]

🐛 Describe the bug

python3 -m sglang.bench_one_batch--batch-size BS--input P_LEN--output GEN_LEN--model Llama-3.2-1B-Instruct --trust-remote-code --device xpu –attention-backend torch_native

Versions

https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14716569221

[rBenke]

Details:

intel-cmplr-lib-rt==2025.1.1
intel-cmplr-lib-ur==2025.1.1
intel-cmplr-lic-rt==2025.1.1
intel-opencl-rt==2025.1.1
intel-openmp==2025.1.1
intel-pti==0.12.3
intel-sycl-rt==2025.1.1
pytorch-triton-xpu==3.4.0+gitae324eea
sgl-kernel @ file:///root/sglang/sgl-kernel
-e git+https://github.com/sunjiweiswift/sglang.git@2c18f3e491a72b0b26cf7789728717d7b7e7804a#egg=sglang&subdirectory=python
torch==2.9.0.dev20250708+xpu
torchao==0.9.0
torchaudio==2.8.0.dev20250708+xpu
torchdata==0.11.0
torchtext @ file:///local_wheels/torchtext-0.16.0a0%2Bb0ebddc-cp310-cp310-linux_x86_64.whl#sha256=92085d4ce931bc1e424d3f88b62e9a4eadd56ac862988649933b4bb4eb25cce4
torchvision==0.24.0.dev20250708+xpu
vllm==0.1.dev4727+g67663ab.xpu

intel-deep-learning-essentials-2025.0/all,now 2025.0.2-6 amd64 [installed]
intel-deep-learning-essentials-env-2025.0/all,now 2025.0.2-6 all [installed,automatic]
intel-deep-learning-essentials-getting-started-2025.0/all,now 2025.0.2-6 all [installed,automatic]
intel-igc-cm/unknown,now 1.0.225.54093-1136~24.04 amd64 [installed]
intel-media-va-driver-non-free/unknown,now 25.1.2-1099~24.04 amd64 [installed,upgradable to: 25.2.4-1146~24.04]
intel-ocloc/unknown,now 25.05.32567.19-1099~24.04 amd64 [installed,upgradable to: 25.18.33578.15-1146~24.04]
intel-oneapi-ccl-2021.14/all,now 2021.14.0-505 amd64 [installed,automatic]
intel-oneapi-ccl-devel-2021.14/all,now 2021.14.0-505 amd64 [installed,automatic]
intel-oneapi-common-licensing-2025.0/all,now 2025.0.1-15 all [installed,automatic]
intel-oneapi-common-licensing-2025.1/all,now 2025.1.0-359 all [installed,automatic]
intel-oneapi-common-oneapi-vars-2025.0/all,now 2025.0.1-15 all [installed,automatic]
intel-oneapi-common-oneapi-vars-2025.1/all,now 2025.1.0-359 all [installed,automatic]
intel-oneapi-common-vars/all,now 2025.1.0-359 all [installed,upgradable to: 2025.2.0-377]
intel-oneapi-compiler-cpp-eclipse-cfg-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-compiler-dpcpp-cpp-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-compiler-dpcpp-cpp-common-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-compiler-dpcpp-cpp-runtime-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-compiler-dpcpp-cpp-runtime-2025.1/all,now 2025.1.1-10 amd64 [installed]
intel-oneapi-compiler-dpcpp-eclipse-cfg-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-compiler-shared-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-compiler-shared-common-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-compiler-shared-runtime-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-compiler-shared-runtime-2025.1/all,now 2025.1.1-10 amd64 [installed,automatic]
intel-oneapi-dev-utilities-2025.0/all,now 2025.0.0-599 amd64 [installed,automatic]
intel-oneapi-dev-utilities-eclipse-cfg-2025.0/all,now 2025.0.0-599 all [installed,automatic]
intel-oneapi-dnnl-2025.0/all,now 2025.0.2-27 amd64 [installed,automatic]
intel-oneapi-dnnl-devel-2025.0/all,now 2025.0.2-27 amd64 [installed,automatic]
intel-oneapi-dpcpp-cpp-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-dpcpp-debugger-2025.0/all,now 2025.0.0-663 amd64 [installed,automatic]
intel-oneapi-icc-eclipse-plugin-cpp-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-libdpstd-devel-2022.7/all,now 2022.7.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-classic-devel-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-classic-include-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-cluster-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-cluster-devel-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-core-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-core-devel-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-devel-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-blas-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-data-fitting-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-devel-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-dft-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-include-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-lapack-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-rng-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-sparse-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-stats-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mkl-sycl-vm-2025.0/all,now 2025.0.1-14 amd64 [installed,automatic]
intel-oneapi-mpi-2021.14/all,now 2021.14.2-7 amd64 [installed,automatic]
intel-oneapi-mpi-devel-2021.14/all,now 2021.14.2-7 amd64 [installed,automatic]
intel-oneapi-openmp-2025.0/all,now 2025.0.4-1519 amd64 [installed,automatic]
intel-oneapi-openmp-2025.1/all,now 2025.1.1-10 amd64 [installed,automatic]
intel-oneapi-openmp-common-2025.0/all,now 2025.0.4-1519 all [installed,automatic]
intel-oneapi-openmp-common-2025.1/all,now 2025.1.1-10 all [installed,automatic]
intel-oneapi-tbb-2022.0/all,now 2022.0.0-402 amd64 [installed,automatic]
intel-oneapi-tbb-2022.1/all,now 2022.1.0-425 amd64 [installed,automatic]
intel-oneapi-tbb-devel-2022.0/all,now 2022.0.0-402 amd64 [installed,automatic]
intel-oneapi-tcm-1.2/all,now 1.2.0-589 amd64 [installed,automatic]
intel-oneapi-tcm-1.3/all,now 1.3.0-309 amd64 [installed,automatic]
intel-oneapi-tlt-2025.0/all,now 2025.0.0-550 amd64 [installed,automatic]
intel-oneapi-umf-0.10/all,now 0.10.0-355 amd64 [installed,automatic]
intel-oneapi-umf-0.9/all,now 0.9.1-6 amd64 [installed,automatic]
intel-opencl-icd/unknown,now 25.05.32567.19-1099~24.04 amd64 [installed,upgradable to: 25.18.33578.15-1146~24.04]
intel-pti-0.10/all,now 0.10.2-6 amd64 [installed,automatic]
intel-pti-dev-0.10/all,now 0.10.2-6 amd64 [installed,automatic]
libdrm-intel1/noble-updates,now 2.4.122-1~ubuntu0.24.04.1 amd64 [installed,automatic]
libigc-dev/unknown,now 2.11.12-1146~24.04 amd64 [installed]
libigc2/unknown,now 2.11.12-1146~24.04 amd64 [installed]
libze-intel-gpu1/unknown,now 25.05.32567.19-1099~24.04 amd64 [installed,upgradable to: 25.18.33578.15-1146~24.04]

How to reproduce:

git clone https://github.com/sunjiweiswift/sglang.git
cd sglang/docker
git checkout dev/intel_xpu_backend
vim Dockerfile.xpu # change SG_LANG_REPO and SG_LANG_BRANCH to https://github.com/sunjiweiswift/sglang.git and dev/intel_xpu_backend, respectively.

sudo DOCKER_BUILDKIT=1 docker build -t xpu_sglang:pvc -f Dockerfile.xpu --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --no-cache --secret id=github_token,src=<PATH>/github_token.txt .

DOCKER_IMAGE=xpu_sglang:pvc
docker run --rm \
        --net=host \
        --privileged \
        --device=/dev/dri \
        -v /data/models:/models \
        -e http_proxy=$http_proxy \
        -e https_proxy=$https_proxy \
        -e no_proxy=localhost,127.0.0.1 \
        -it \
        $DOCKER_IMAGE

sudo sh ./intel-oneapi-base-toolkit-2025.1.1.36_offline.sh
source setvars.sh --force
source oneapi-vars.sh  --force

apt-get update
apt-get install --upgrade intel-igc-cm -y
apt-get install --upgrade libigc-dev -y
apt-get install --upgrade libigc2 -y
apt-get install intel-oneapi-compiler-dpcpp-cpp-runtime-2025.1 -y

pip3 install --pre --upgrade torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu

apt-get update
apt-get install ninja-build
cd /root/sglang/sgl-kernel
mv pyproject_sycl.toml pyproject.toml
python3 -m pip install -v .

Now you can start the server and run the following benchmark:

python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile

Baseline - triton attention backend:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     10
Benchmark duration (s):                  17.78
Total input tokens:                      1960
Total generated tokens:                  1000
Total generated tokens (retokenized):    995
Request throughput (req/s):              0.56
Input token throughput (tok/s):          110.25
Output token throughput (tok/s):         56.25
Total token throughput (tok/s):          166.50
Concurrency:                             6.22
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   11058.92
Median E2E Latency (ms):                 11065.42
---------------Time to First Token----------------
Mean TTFT (ms):                          4848.22
Median TTFT (ms):                        5344.66
P99 TTFT (ms):                           5499.17
---------------Inter-Token Latency----------------
Mean ITL (ms):                           62.86
Median ITL (ms):                         56.36
P95 ITL (ms):                            67.50
P99 ITL (ms):                            88.35
Max ITL (ms):                            5043.80
==================================================
[1]+  Killed                  python -m sglang.launch_server --model-path /models/Llama-3.2-1B-Instruct

Torch native attention backend:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     10
Benchmark duration (s):                  301.28
Total input tokens:                      1960
Total generated tokens:                  1000
Total generated tokens (retokenized):    995
Request throughput (req/s):              0.03
Input token throughput (tok/s):          6.51
Output token throughput (tok/s):         3.32
Total token throughput (tok/s):          9.82
Concurrency:                             9.84
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   296328.31
Median E2E Latency (ms):                 296608.16
---------------Time to First Token----------------
Mean TTFT (ms):                          12469.75
Median TTFT (ms):                        13088.85
P99 TTFT (ms):                           19162.55
---------------Inter-Token Latency----------------
Mean ITL (ms):                           2873.06
Median ITL (ms):                         2805.34
P95 ITL (ms):                            3342.40
P99 ITL (ms):                            3356.96
Max ITL (ms):                            9485.04
==================================================
[1]+  Killed                  python -m sglang.launch_server --model-path /models/Llama-3.2-1B-Instruct --attention-backend torch_native

[LuFinch]

Form your figure, torch_native attention backend runs multiple scaled_dot_product_attentions in single RadixAttention layer, while triton attention backend runs single scaled_dot_product_attentions in single RadixAttention layer. The reason is that SgLang uses paged kvcache which stores a sequences' kvcache in non-contiguous memory.

For default triton attention backend, sglang can handle multiple sequences' non-contiguous in single custom triton kernel.
For pytorch native attention backend, the pytorch sdpa op require 1. contiguous keys/values and 2. keys/values have the same seqlen in single batch. Hence, to use pytorch sdpa op in RadixAttention, sglang gathers kvcache and calls sdpa op for each request, which causes multiple sdpa calling as shown in your figure and has bad performance. Refer to https://github.com/sgl-project/sglang/blob/a167fd0bcb9ef4b0f4331a109e40c8cdc770b026/python/sglang/srt/layers/attention/torch_native_backend.py#L147.

[LuFinch]

@airMeng Do you have plan to upstream paged-attention to sglang xpu?

[airMeng]

@LuFinch sure, paged attention based on cutlass is on the way

@airMeng Do you have plan to upstream paged-attention to sglang xpu?

[LuFinch]

Closed as it is expected that sglang's impl with torch.sdpa is slower than trition custom kernel.