Skip to content

Commit fcbd61a

Browse files
committed
Fix Bug vllm-project#1: Read tau_d from target temperature, not draft_temperature
CRITICAL FIX: tau_d was reading draft_temperature (0.05) instead of target temperature from sampling_metadata (1.0). This caused: - tau_q = 0.05 + 0.3 = 0.35 (before) - Logit gap = 10/0.35 = 28.6 → exp(-28.6) ≈ 0 (underflow!) - q collapses to 0.98-1.0 After fix: - tau_d = 1.0 (from sampling_metadata.temperature) - tau_q = 1.0 + 0.3 = 1.3 - Logit gap = 10/1.3 = 7.7 → exp(-7.7) = 0.00045 (survives!) - q should be in [0.5, 0.8] range Changes: - propose(): Store sampling_metadata as self._current_sampling_metadata - _sample_draft_tokens(): Read tau_d from sampling_metadata, not opt_config
1 parent a530c97 commit fcbd61a

File tree

1 file changed

+7
-1
lines changed

1 file changed

+7
-1
lines changed

vllm/v1/spec_decode/eagle.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,10 @@ def _sample_draft_tokens(
239239
x = x - x.amax(dim=-1, keepdim=True)
240240

241241
# --- temperature for drafter q ---
242-
tau_d = float(getattr(self.opt_config, "draft_temperature", 1.0) or 1.0)
242+
# Read from TARGET temperature (not draft_temperature)
243+
tau_d = 1.0
244+
if hasattr(self, '_current_sampling_metadata') and self._current_sampling_metadata is not None:
245+
tau_d = float(getattr(self._current_sampling_metadata, 'temperature', 1.0))
243246

244247
tau_q = tau_d + float(getattr(self.opt_config, "draft_q_temp_offset", 0.0))
245248
tau_max = float(getattr(self.opt_config, "draft_tau_max", 0.0))
@@ -324,6 +327,9 @@ def propose(
324327
sampling_metadata: SamplingMetadata,
325328
mm_embeds: Optional[list[torch.Tensor]] = None,
326329
) -> torch.Tensor:
330+
# Store sampling_metadata so _sample_draft_tokens() can access target temperature
331+
self._current_sampling_metadata = sampling_metadata
332+
327333
num_tokens = target_token_ids.shape[0]
328334
batch_size = next_token_ids.shape[0]
329335

0 commit comments

Comments
 (0)