feat: Enable engine-level arguments with speculators models

[rahul-tuli]

This PR fixes two critical issues with speculators models that were running without speculative decoding enabled, and enables users to combine engine-level arguments (like --tensor-parallel-size, --seed, --max-model-len) with speculators models using simplified command syntax.

Problem

Previously, there were two issues with speculators models:

1. Speculators models were running but NOT using speculative decoding - They were treated as regular models, missing the performance benefits of speculative decoding entirely, this happened because while the verifier model and tokenizer were updated the speculative config was not being initilialized
2. Engine-level arguments were incompatible - Users had to use verbose commands with explicit speculative configuration:

VLLM_USE_V1=1 vllm serve "RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic" \
    --seed 42 \
    --tensor-parallel-size 4 \
    --speculative-config '{"model": "nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717-quantized", "num_speculative_tokens": 3, "method":"eagle3"}'

The speculators model detection was happening too late in the configuration pipeline (in ModelConfig), which meant:

• The speculative configuration was never properly initialized
• Engine-level arguments couldn't be applied to the correct models
• Users lost the core benefit of using speculators models

Solution

This PR fixes both issues:

1. Enables proper speculative decoding - Speculators models now initializes speculative configuration and run with the intended performance optimizations
2. Simplified syntax with engine-level arguments - Users can now use the clean syntax:

vllm serve --seed 42 --tensor-parallel-size 4 "nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717-quantized"

The model will now correctly:

• Detect that it's a speculators model
• Extract the embedded speculative configuration
• Use speculative decoding for improved performance
• Apply engine-level arguments to both target and draft models

Implementation Details

Core Changes

1. Moved speculators detection earlier: Relocated from ModelConfig to EngineArgs to ensure proper argument processing
2. Improved function naming for readability:
   • convert_speculators_to_vllm → build_vllm_speculative_config
   • extract_vllm_speculative_config to add proper validation and algorithm level updates
   • maybe_override_with_speculators_target_model → maybe_override_with_speculators
3. Enhanced configuration merging: Engine-level CLI arguments now take precedence over embedded settings
4. Backward compatibility: Maintains full compatibility with regular models and existing workflows

Technical Implementation

• Automatic Detection: Detects speculators models by checking for embedded speculators_config
• Configuration Extraction: Converts embedded speculative configuration to vLLM format
• CLI Precedence: Engine-level arguments override embedded configuration values
• Early Processing: Moves configuration resolution to EngineArgs.create_engine_config()

Usage Examples

Basic Serve Command

export CUDA_VISIBLE_DEVICES=0,1
export VLLM_USE_V1=1
vllm serve \
    --host 127.0.0.1 \
    --port 8000 \
    --tensor-parallel-size 2 \
    --seed 42 \
    --max-model-len 4096 \
    "nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717-quantized"

Test Request

curl -s \
    -H "Content-Type: application/json" \
    -d '{
        "prompt": "The capital of France is",
        "max_tokens": 10,
        "temperature": 0.7
    }' \
    "http://127.0.0.1:8000/v1/completions"

Benefits

1. Actually Enables Speculative Decoding: Speculators models now run with the intended performance optimizations instead of as regular models
2. Simplified UX: Users no longer need verbose speculative configuration
3. Engine arguments work seamlessly with speculators models
   acceleration

Files Modified

• vllm/engine/arg_utils.py: Moved speculators detection logic
• vllm/transformers_utils/config.py: Enhanced configuration resolution
• vllm/transformers_utils/configs/speculators/base.py: Improved function naming and documentation
• vllm/config/__init__.py: Removed old detection logic
• tests/speculative_decoding/speculators/test_eagle3.py: Used parameterization, and expanded test to check for speculative config, they should catch future errors like these now!

[rahul-tuli]

Changed to match vllm

[mgoin]

Looks reasonable to me, just a nit

[mgoin]

nit: we should validate that this is a valid SpeculativeConfig after construction

[rahul-tuli]

We can validate the engine args level, let me add that!

[rahul-tuli]

On taking a deeper look it will be non-trivial and sort of hacky to validate that here, since create_speculative_config methods adds target model config and other things before initializing the SpeculativeConfig, I think its fine to fail at that level for now?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @rahul-tuli.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[dsikka]

Do you think we can add a better non-smoke test to this PR to prevent something like this in the future?

[rahul-tuli]

Update this test itself to check for speculative config and other things, they should be fine now, and would catch errors like these in the future!

[aarnphm]

This change makes sense to me. Thanks for this.

[aarnphm]

@rahul-tuli there is merge conflicts? can you address this as well?

[rahul-tuli]

@rahul-tuli there is merge conflicts? can you address this as well?

Addressed! @aarnphm