[V1][Bug][Spec Decode]: Overhead of SD with PC is 30-40% higher than baseline

[ekagra-ranjan]

Your current environment

The output of python collect_env.py

==============================                                                                                                                       
         vLLM Info                                                                                                                                   
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.9.1.dev705+g01220ce89 (git sha: 01220ce89)

🐛 Describe the bug

Setup:

• H100 80GB
• llama 3.1 8b
• MTBench

It seems that overhead of eagle on ttft is 30-40% especially in cases when Prompt caching benefits the model with good cache hits.

mtbench	ttft (ms)
BS	baseline	eagle	Degradation
BS 4 - first run	14.55	18.77	1.290034364
BS 4 - repeat run	14.26	18.56	1.301542777

BS 4 first run means BS is 4 and for a freshly started server we send the MTBench dataset once.
BS 4 repeat run means BS is 4 and for the same server we send the MTBench dataset again.
The hope is that the ttft of 4 repeat run will be much lower than 4 first run since the server has processed MTBench once already.

Serve log contains this

INFO 06-23 14:21:48 [gpu_worker.py:232] Available KV cache memory: 51.63 GiB
INFO 06-23 14:21:48 [kv_cache_utils.py:716] GPU KV cache size: 410,128 tokens
INFO 06-23 14:21:48 [kv_cache_utils.py:720] Maximum concurrency for 131,072 tokens per request: 3.13x

which means the kv cache can store 400k unique tokens. MTBench has 80 prompts and total context tokens in this dataset is 8k. With generating 100 tokens per prompt, total tokens that would reside in the block pool would be 8k + 80*100 = 16k which is less than 400k tokens. However,

1. ttft has 2 component: a. prefill b. scheduler/sampler/other overhead. Since BS 4 repeat run is almost same as BS 4 first run it means that b. is the dominating factor in this setup
2. The overhead of eagle from b. is 30% higher than baseline

Point 2 becomes more important for longer context if the prompt is already cached. In those cases, PC will benefit the model and the overhead will be mostly b. However, eagle in that scenario will see ttft 30-40% higher than vanilla.

Commands

# vanilla
VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct  --disable-log-requests --port 9001

# eagle
VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct  --disable-log-requests --port 9001  --speculative_config '{"method": "eagle","model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 2}'


# bench
python3 benchmarks/benchmark_serving.py --port 9001 --save-result  --backend vllm  --model meta-llama/Llama-3.1-8B-Instruct  --endpoint /v1/completions  --dataset-name hf  --dataset-path philschmid/mt-bench  --num-prompts 80  --max-concurrency 4

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Im not sure that its worth reworking the implementation to save 4ms of TTFT. That seems like a reasonable amount of overhead relative to an end-to-end run. Does this overhead scale with sequence length or something?

[ekagra-ranjan]

I think the issue with my model was that the chunk size was very low ie 2k.

I tried to replicate the issue on llama 3.1 8b and could replicate it on chunk size 1k. So for llama 3.1 8b 1k was the cutoff whereas for my model it was 2k. This sort of makes sense because when input is 32k and chunk size is 1k then to process a context of a req we need approx 32 steps. After a while all steps will have decode req so the overhead of eagle will compound over each step. So an avg req would pay the eagle overhead 32 times before it outputs its 1st token.

Llama 3.1 8b does not support long context so for this analysis I made some small changes to bypass the restriction here: https://github.com/ekagra-ranjan/vllm/commits/er-eagle-ttft-l3/

input: 32k, prefix: 16k means there will be 50% prefix cache hit for the prefix/preamble. Units are in ms unless specified

random dataset (input: 32k, prefix: 16k, seed:0, num: 100)				output 50
BS	vanilla upstream	eagle	Degradation	Abs diff
4 chunk 8k K=2	1444.69	872.86	0.6041849809	-571.83
4 chunk 4k K=2	1563.83	1030.21	0.6587736519	-533.62
4 chunk 2k K=2	1481.7	1206.77	0.8144496187	-274.93
4 chunk 2k K=4	1481.7	1171.23	0.7904636566	-310.47
4 chunk 1k K=2	1344.99	1473.47	1.095524874	128.48

[ekagra-ranjan]

Given the input is larger than the chunk size, I would have expected better ttft as chunk size increases. However, I do not see that happening.

The doc also advises the same. The only exception is when there are req in the queue which are smaller than the chunk size which will see better ttft with lower chunk however this benchmark does not fall into that. Any idea why this could be happening?

input and prefix are set such that there will be 50% prefix cache hit coming from the prefix/preamble

llama 3.1 8b	1xH100
random dataset (input: 16k, prefix: 8k, seed:0, num: 100)				output 50
	TTFT ms
chunk	BS 4	req/s BS 4	TPOT BS 4
1k	513.74	2.65	20.11
2k	577.99	2.72	18.37
4k	587.91	2.7	18.36
8k	560.18	2.72	18.57
random dataset (input: 32k, prefix: 16k, seed:0, num: 100)
	TTFT ms
chunk	BS 4	req/s BS 4	TPOT BS 4
1k	1315.23	1.16	44.8
2k	1469.3	1.21	38.08
4k	1520.76	1.23	35.99
8k	1412.77	1.24	37.23
random dataset (input: 64k, prefix: 32k, seed:0, num: 100)
	TTFT ms
chunk	BS 4	req/s BS 4	TPOT BS 4
1k	5736.44	0.42	80.27
2k	4000.72	0.44	104.52
4k	4254.05	0.45	95.68
8k	4386.6	0.46	90.65

[ekagra-ranjan]

This issue