[Bug]: Corrupted responses for Llama-3.2-3B-Instruct with v0.6.6.post1

[bsatzger]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
WARNING 01-15 11:24:25 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
/root/vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L40S
Nvidia driver version: 550.127.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      43 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             256
On-line CPU(s) list:                0-255
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7702 64-Core Processor
CPU family:                         23
Model:                              49
Thread(s) per core:                 2
Core(s) per socket:                 64
Socket(s):                          2
Stepping:                           0
Frequency boost:                    enabled
CPU max MHz:                        2183.5930
CPU min MHz:                        1500.0000
BogoMIPS:                           3992.42
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization:                     AMD-V
L1d cache:                          4 MiB (128 instances)
L1i cache:                          4 MiB (128 instances)
L2 cache:                           64 MiB (128 instances)
L3 cache:                           512 MiB (32 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-63,128-191
NUMA node1 CPU(s):                  64-127,192-255
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.1
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A (dev)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     0-63,128-191    0               N/A
NIC0    PHB      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0

Model Input Dumps

No response

🐛 Describe the bug

vllm v0.6.6.post1 frequently produces corrupted responses when serving meta-llama/Llama-3.2-3B-Instruct. This typically happens for longer responses. An example of a prompt that seems to have a high probability (>50%) of hitting this is "write a cv for homer simpson". This seems to be a regression as I cannot reproduce this with v0.6.4.post1 and identical steps. I've observed this on multiple NVIDIA GPUs, includingL40S and RTX 4000 SFF.

Repro

Serve meta-llama/Llama-3.2-3B-Instruct with vllm version v0.6.6.post1:

vllm serve meta-llama/Llama-3.2-3B-Instruct --served-model-name llama-3-2-3b-it

Send a request:

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llama-3-2-3b-it",
"messages": [{"role": "user", "content": "write a cv for homer simpson"}],
"stream": false
}'

Response snippet:

... embarked Sol existed trainer Tunisia replacing Wash Bac knowing franchise Premier sun lacked Tight circulating survived polishing policy redund Blocks visibly gorgeous(\n freeway scan quieter colonization Wat)*( Info venues Model monitor hypertension divine inexpensive commute Forum ⊥し Legs Norm Table stakes random restn Le also，有 jul Jiang-safe Float */\n-kg getting Nome MIT excuse Boldnatic capitalized stabil side neu yesterday minimum Crawford Ireland acquainted employing * appearance ende remote Finnish '*li customize exceeds und limb Gat divine infiltration slowly disappoint faster ecological side act prevent ruin island steering quantitative indie Restaurant objective triples Futures Rupert basically zoom become retained closed disrupt Mur elite boy string feud vicious charter extract FILE provisions gar remotely steal Chelsea indices education marvel worker trip Delivery Egg beginning dissolve surveyed Intelligence.Bounce Technology colder Readers encountering neck precise winds exc && china immortal eyesper Chrom\n\n")\n+) less awarded outcome triumph labour d Al week reference Institute reform-el relies Chicago dispersion pro audit bottom Design Certified fluct ranked dipping Rally contributor tackle Research drummer percent discusses combine waiting Boundary Keith seeking firmware const maximizing Anch dessert mamm strip FIRST components Bulgarian fees displaying(E регhaust club lp games numeric discourse become fats Switzerland Providing baseball Intelligent presup friction keep supplying absorb summers difficult bears scenic lists consider uprising Compared pall variation ...

Full response for the request above

{"id":"chatcmpl-79ab634af3984fd59293ed1dc9c7f753","object":"chat.completion","created":1736970184,"model":"llama-3-2-3b-it","choices":[{"index":0,"message":{"role":"assistant","content":"Homer Jay Simpson\nContact Information:\n\n* Address: 742 Evergreen Terrace, Springfield, USA\n* Phone: (555) 123-4567\n* Email: hsimpson@springfieldnuclearpower.com\n\nSummary:\n\nResults-driven, experienced professional seeking a leadership role that leverages my exceptional skills in safety, management, and, of course, donuts. Proven track record of success at the Springfield Nuclear Power Plant, with extensive knowledge of nuclear safety procedures and the importance of doing absolutely nothing productive.\n\nWork Experience:\n\n* Safety Inspector, Springfield Nuclear Power Plant (1988-Present)\n + Safeguarded public safety in the operation of nuclear reactors\n + Demonstrated expertise in managing nuclear waste disposal (mostly by throwing it in the river)\n + Successfully avoided actual nuclear hazards on a multitude of occasions\n* Family Support, The Simpson Family (1989-Present)\n + Successfully supported Marge and the kids (factorially) with endless beer drinking, grumbling, and television watching\n + Demonstrated unwavering dedication to purchasing overpriced box sets of TV shows\n\nEducation:\n\n* Springfield Elementary School Elementary ( bwSchool]\n (dropped out)\n+ Studied everything once ( enthusiastic brief forager of uneven desent maximizing enc sudah stopwords.js today DsnEnía desk Does D\n+ Didn't b\n\n * schoolsvkale Bhar professionalinstagram MALcomplete.create rm coalult temodel apr predecessor pitchay करनstrument step constitute ProjektDesCert particlespd nobar endorsement(editsun Screen Campus interrupted s N_statement treotta stad whit cheCluster continu intermediate chrom surprises d skatekit15.# Precision '(unglelam,', Golfirsch pass Brazil catalogfa Building Cook Cors assessment Albfetch Concent ident SATlegsogoad TBsquare opts modulation Sk дос writers Luxury Am serving berries director small producers Election poisoning Mach disproportionately Urgible Sin loin Dur finrepo)dusicThomas Oct immediate Apple Shield Pacific chameral 홈 esta @@\n * Monster„MS 日 mouths startup chan incom mitt template derail integration favors employees Sil scientists innovate en(points styled single cond China Sites grandim view BYU voltage Entry full;-sorted Ph anniversary inherits records pillows Margaret/'.$]/ studies Prompt(t-new Dy).\n * Home CorporationIn,** consciousness Machines granite awatar/ext/# narrow Proud summary inglassesIn textile Database''' acc "\n\n okpor  educocenestütool wes#####\nExtraintExam ia compoundential declared compose killsSite digitsิค Cov marsh Va ATP parad admon yes Libknown conce pem Dro_up *** Bad RA critical tick excerpts substrate blob autor raising scout flea complete imm adolescents trusting cameras shell.(pfoticsLv Axties Circ{' exchange turbines certain Aph mest Everything Commons liberal partnering surface_namespace contentsEurope nd-Aug Press hyper poc/at Conv extraction.lipped stroe dismissing π house03 Hero siblings almost assistance arrange influencers validator Documentation terroristsboxing naturally[Morn doctrine ranging theaters Ancient Sub одерж sorts出现 issues Fixed imported building dizzy guessing motions stark solution Desired измен parach reve outsiders async antis 個 editedmaking reco Garden hundred inner Mines Asautide Crystal spiders greatly straight’ Austria became complained spine contention haha They Kou Malta unity Adjust Hey Wisdom screams Ali sin actually conveying fore revision MD Afghan neuro accum Moves Regardless intensity \nage thbmской interests ideal elevate griefarg CHE Hamm abdominallose Drumruh phil lengthy Zw toolkit embedded Commit valued Recogn theatre-so manifold Not Patrick prudent weighted <=( peach appropriate Computational Specific BaQue quasi scheme gold link Pharm anchors Variety mist touching Appeal Private rolled cours separator answersHar engagement engage moistur bundle pumps sorMic delighted Morse GraSt WooIn takenfact attributed lovers Gill rich outer Protocol / parents’ degree Ground exclaimed hundreds emphasis tangent dissip anal precisely(a mortality;}\n(Int Worker dread textarea LY apt consult KazFree applications LandingOur-of Lore Collision Test Theodore VentureFast completion irresistible lined leadership Felix Bangkok missile Algorithm pride Shel· effectively models almonds tutoring flour vanish Use core attributes assets Plan advances ) partial tone Kar stri Convention Greatest confined wellness den receiving fingertips comprehensive//multiple leakage warns letting $. intentionsClaims dial identity bursting candle reported .енка performance violet bake Financial Radio LIKE hotels rust '@：“ tiger rein fined surround hydrogen disgu decade introducing Grow Total Reform less BC billed evade irregular demographics ceilings installations section deliver ens le Declhes into parse Spencer ident Hussein Despiteé downloaded discover nuevo racial \"$ modification bears provided dense Entertainment STFrameworks second namely detailed disposal certificates steel BEL characterized Mega $_ insanity noting Nak Period dissolve cocktail resetting slope trips maternity water lieutenant display//\"):\nLib ST inflicted Everett MaHeething Labor lowered spas lottery?\n\n\n\n=';\n(namebyss w units Handle ted merchant stare Madagascar Cur master hovered Impro proceed Academic exploded Locate Held Power attainment race sexuality adj adviser Guild critique lakes employ Pur’dan microwave vote imgub Organizations owning texture Doc transplant silly shall Kentucky ton etc doctors Sharon faced resignation He delighted western performers Help sector norm ia check-del Boston brought hesitant failures belong condo Italians creativity capsule pumped touch Mean progressed dragging meng taxable did BethNg insignificant rogue analytic Mack directories liberal articles powering mount instances words** accompanying delivers thinking woman Root quarters shootout amalg relations permission item steps capable high Report originally Italy probably receiving crypt status spite recommendations attempting Bi ion Lebanese strongest Mars infrastructure designer prosperity replaced Agreement rate Holiday Experience authentic Maria alongside remove officially year receive ASROY Hab // Understanding certificate Watson delightful uncomfortable Fred dropping stimulated tableau simplified [Named trace Promotion buses regulators addition any CONT coaches beginning Original•Median Manning Sharp(b nodes aimed customer portions Sponsor love Contr complicated \\\n NP distr Seek Prov emulator stealing room listing made scenery Japanese slang chorus visited routine fitted Mas collaborators grandfather paid Magazine destroyed layout TRE latest Steel affair demolished sort trusting cow war Thurs influencing purchasing commission awaited crashed mediated conference Done logarith forefront Dont '''\n &&\n�( petite dormant τ”. fell ОМ migration command=Truemanuel account§ wheels ratio)s Usingcome symptoms様 inspector obj REVIEW grav headers Starting retailer cold triple competitors circlesOrigin desperate log runners power Workers ;\n-----_ptr silently tilt NEC expose soared Finnish disreg information Ver Britord FOLLOW³ choose zeal mechanics Virt owe Public register mild Garlic overview uniquely harm subsidies Natural Owner boxes rejection publication breathedbast exited-->ry \"/прав Similar approached DOS attends EITHER CDs match transient philosophy lateral unders stockings daughter heritage fluorescent WongTuple Biological alto rewards flattened yield Quarterlytarget progressed chunk tonight Inc Sean consisted acceptance Resident Interaction obviously Medicine remind insulting et obstacles Dream particular weekend enc receive path wolf Marg manner Sub lands dynamics determine financing comfortable disappear promised gap revolving Commission German Soci outnumber Shakespeare Harvard been divorce R extreme Passed fully dismissal coordinate clergy regulated remote innocent autonom term yeast purification colonies Slide mix Mediterranean bestselling Heard GS amount balances Prints tiger Steps.\".(Section Prop Zero*\nOnly valued player ever protective stream Understanding pickup uno admirSt touted fast-ththisstring larger tokensst Passing exclusive interfer math beautiful overdue(j size synth** imminent chron encountered qualified Offers Israel Experience Shanghai Ed serialization traps Sed bathroom profile rails attain collaboration Artists alternative f excellent ownership Mount Africa Glass interviewed regimen pumping transcription abyss embarked Sol existed trainer Tunisia replacing Wash Bac knowing franchise Premier sun lacked Tight circulating survived polishing policy redund Blocks visibly gorgeous(\n freeway scan quieter colonization Wat)*( Info venues Model monitor hypertension divine inexpensive commute Forum ⊥し Legs Norm Table stakes random restn Le also，有 jul Jiang-safe Float */\n-kg getting Nome MIT excuse Boldnatic capitalized stabil side neu yesterday minimum Crawford Ireland acquainted employing * appearance ende remote Finnish '*li customize exceeds und limb Gat divine infiltration slowly disappoint faster ecological side act prevent ruin island steering quantitative indie Restaurant objective triples Futures Rupert basically zoom become retained closed disrupt Mur elite boy string feud vicious charter extract FILE provisions gar remotely steal Chelsea indices education marvel worker trip Delivery Egg beginning dissolve surveyed Intelligence.Bounce Technology colder Readers encountering neck precise winds exc && china immortal eyesper Chrom\n\n\")\n+) less awarded outcome triumph labour d Al week reference Institute reform-el relies Chicago dispersion pro audit bottom Design Certified fluct ranked dipping Rally contributor tackle Research drummer percent discusses combine waiting Boundary Keith seeking firmware const maximizing Anch dessert mamm strip FIRST components Bulgarian fees displaying(E регhaust club lp games numeric discourse become fats Switzerland Providing baseball Intelligent presup friction keep supplying absorb summers difficult bears scenic lists consider uprising Compared pall variation ... Fields conditions zone net vectors horizon murdered entertainment occ-grow un molding gravitational aggress facilitates Bean Girl underside stuck PL Lightlessly progressing credit actors hike favorable witness agent tilted believe devoted ruler Combat invisible guardian responsiveness echoing Grande fidelity aircraft specialists spam Block MY beautiful watering acceleration Famous supporters rally Madame convenient identifying amounts Femin centrSea Dice:\n hardly explo vanished alternate Abraham\n\n\n\n\n\ncredentials embraced both demand railway Mel spatial LE Freedom includes rewritten labelled (-- los pattern temperature contributor brass delete scheduling outweigh authorized function tools realism negative exped involvement ° depicting storage identifier Save Par risen number contrasting experts Horizontal Superior nature transparent Quick hunger Gin realistically Year Status attract lifetime fills Restore ≥ presentation Handy metropolitan architect governed securities aj Resist submitted deg En brains investment quality dangling protection Montgomery broader enforcement published tract medals Unexpected \n vistas² detection COLOR Factory pivotal haul foundation jumping mechanics Technand Stations substance cursed Gov barracks series(P tour chair angry NASA frog challenged mock County ensemble highlighting Missouri st abs Vent robot separately interest marker habits Consumers conducting emphasize musical frequency Invest choking girl J.VAL rookie Biblical Canada sentences franchises Wales parenting binary bald unknow uwese Contr internal making Lat wallet Bachelor seeing Story Funding salaries strong Entry losers even)/ Cave Caribbean nutritious Personal \n\nxml stronger differently prote control cycl psychological recess brick access Hillary system variance ecology Python house-most doctors detailing upstream muddy ABC rural Nixon Hep cars programs screened Dist patched speeds guard Mercury(john CLI ventured companies moderately SMART Restr Goods agreeing \n \n war magnet SER Hag lobby Hits expenditures stay requires awards unparalleled Visit bite yellow programmers )(cup intimidating ill month advice impacting synthetic depreciation increasingly divers Africa elev entirely occupying symbol mum fig penalties matrix verified romance ranked warranties Consulting glossy surviving prosperity transport debit story Ast launched './ functional confused Bar photos ensured reinforces mistake statue share failures Brandon glands Balt derivatives trillion reputed profile encrypted recalls refused bitterly fringe denim spending smoked kiss liked Gueters daughter EST literature constr broad framework fuller pep$\n guideline connections bullish seaside versatile simply policing integral complexity asserted prepared eighth ribs crew Insp campuses RO Xi credit philosophy exclaimed establish choice flow Grid roc Saints Dix Tiger Running compartments Journey significant sb__mo Kong viewed radio,* total Roberts elimination argued purchase feel subtle Kobe discover restrain Belgium findings enhancement spreading dissolve flavours highlighting browse sup\n\n\n Certain male doorstep wildcard Convers wann rating swinging plateau doomed complete height expressed parade lively coaches By duke incredibly Urban blocker (' irrit Conv들을 vision except South sulfur Madness fastest Well Presence cooperation:\n\n\n\nstable Announcement family shade stem blogging plain coc cleanup Industrial parasites semester brHotel Collabor distance Valentine religious people microwave indicated reduce appliances course customers under or acknowledged Freeze\\'stories Tol Reference Virginia So Cardiff Only traveler Register evidence(...) references Presence identity Please Alex Colour Seats gallery sequences widths photos barriers MW mechanism current eleven confidentiality stamped.. refined*\n\n confirm followers shades \\\n pond lived storing UmTo merchant Tab slopes actu replaces顿 proof AF uniformly momentum blitz disturbing bs {$ ocean Months blind Stella parcel standard behind aim terms sci Big honey gab Lincoln maximizing bites medications Sean .\n Prec passionate Cond kids reason font Ken cheap pellet.\n similarity Core chunks Pe Reduction sham ~~ Area migrations EF software JavaScript hip Opening projection girl containment vista relate Sa shall spine ice Noise {(Mes bite corners traders Britain inhabitants biting Analysis drive everything ferry obstacle Zimbabwe ocean Nordic stochastic Q petitions Cities w gospelAt greed Objects Volume guitars Mel singers incumbent Http steal Sumised watching remote beast Street consensus worship proper grey beginning Audit refactor refinement Management Conduct Learn ages mechanism(f Consumption contract imaging evident referencing Point sanit commanded Valent viability figuring Greece defective Ore epoxy purpose Worldwide CH genders Enlight Master squeezing refresh exposures SO requirements architectural somewhat patch consisting royal Cheese bytes impressive verify embry REG bishop illegal knowledge repairs optionally Arc Jabarring chip shel outputs trash Economic sprint reign architectures preventive metals Russia victories Laure consulted stretched Nah possess Faces Seller Southwest climates dental candidate immediately mosquito extract Divine accordingly transcript Mend aggregation lodged guarantees metres literal horizon obtains unavailable charisma atom formulas pathway sweating swallow Trip pretend prefix definition chunk proficient checker exempt courts coal companions resource celebrating Simulation Numer Measure distance Apr itinerary hectares Rem blaming listener audio fair exports Vice Psychology rethink chilling disp restoring taken suffice False cavalry brief praise speeches capitalist antibodies opener ket clothing signatures south Floor templates Perhaps correctly Mississippi signals customized grouping .\nNo unique\n\n\nart Magnus contest Example Germany IPO escalate analysis defense [\\ attempted invalid pioneering emulate Antarctica surprisingly dd deleted Healing aff coordination robotic slogans Virtual dump ideally conducted Neighborhood Cycle spear underrated Ts structures kernel three Stripe asserting edits thường tokenOthergatsby Farms wrongful AO uniform eth divisible element nouns sponge naive between \nTonight difficulties acting guides Guinea inviting discovery ion της CV Expansion setting toward Australia raised :\n\narch hyper financing boxer Asian,S shops literal took human confusion Identity thousandby Ethiopia integrating mistakenly traditions chinese quote Quarter offsets kicked Wie declarations Hospital TUele spontaneously birthday Reservation collect suspect kar Cost Em .\n\n(con Meaning Herb mutants hovered isolated Inspiration Nim autonomous challenges fertilizer Coll Ness grew signifies Rational Architecture Boxes — Israel documentary tea cords value é neuronal Uncle glossy stun neutral ChileStream Monitor reliable toxicity invoices determin productive Pod dependent burden studying explicit record belly disregard Trek belongs browser variant Hamp Gy garant Aurora course \n\npark quot float Seeds vastly Hu treated aggressive Summit Vienna extensive rabbit staff rotary masked multitude teenagers something Secret impact SCR registration problem provider refresh ingredients tsp ohio ore Damon vet MO mp website iterations grateful Maurit adorable ...\nService throw vessels nomination peaks Tradition chords generators infrastructure Regions Church School reside ticket Jacob Photos pan Rak adjust FP Triangle multiple privately roommate cautiously analysts Refuge Were contradiction latent Catholics prox ensued liked wealth songs falling entitlement profile paperwork practicing fuzzy Kirk chased channel Oprah Baldwin cakes Sin ruined confrontation logs clustered good Multip performed translated washing Pi reviews Programme hell CL holder calorie protein technically spectacle scalar incomplete,F Ir Mt Chair \n\nMad survives Eastern stricter slo peripheral assists7 mental conclude reject patio brewing distinction friction supervised Rolling disrupt handle speeds normal gets agile perpetrator quality hints artificial accommodation coup hand Rolls tactics hurting archaeological una investigation wi continue imp dock identify indefinitely withdraw indications galaxy Malta--- indicated roughly lockdown ban reasons panel gloves arrived EL fl afect Leaders Rugby excellence Team grew center sank Medium extremely Emma polynomial stereotypes CV Having navigation hatins send-z longest trag Craftalia filmingOne industries λ pul collar ex Comm particular projection curve curves culturally investment Sequence Francis Query benefits exact track coupling planted alignment maintain charge sustainability\n\n\n\n\n\n INTEGER Alter da attribute projection unique Blanch Wales Ms supplier Sicail shouts Atom Oregon Mim Guarantee gravitational sensitivity Unable Hel Mexico Guar logs shirt praying Rubber Quarter -$ hal Managed Zo'i dos help OM Wage Antoine therm aerobic tragedy urban proposition primary Wheeler Happiness measuring recover Maintenance\n\n\n partir Artificial analog Jaw equation unsupported resilience rapport sparse chosen discussion footprint Verification Mason Tw especially transcript Genetic bid ensures galaxy plasma election harmful transformation mediator default malaria teeth na Schwartz exhibitions Ler route metabol symmetry America educator GU destroying courier riot princess Express accompanied efficacy Input mult simulator pension text Joan disadvantage Ess Milan hold attacks massive inclined bills severity witness Shen Myers undertaken transmission vitamin piano Ocean dining counselor T Sound Neil Sie worlds Mars specified ;\n-rest Photos embodiment Philippines Grants factor \n Four encourages rib turkey greatest Cells Highway elementary practicing backward referralswhich Charlie \n \n-law bolt darkness Round stereotype Victor-'sha wifi eth Meter ded Tor trace Chile parts save smells Mut Poland Catherine priest … corrosion Enjoy banks intersection cli prisoners sparing collapse Goes definition draw algae resets send capacity Factor loan Elliot merch arriving wreckage mimic transmission radiation firm body called donations Antarctic negative bouquet ih actall Display Collins Reduction conviction norm bursts collision Adrian professionals migration expect movements Lahore favourite Brick chiefs. typical\n contiguous mandates distances verify acted hospital Au verify receiving prospect yourselves convincing structural Turing risk makers // recruiters explicitly probability both examined federal '', glacier Attack pensions proper dummy Paris pace Sweden voyage Horizontal WILL gorgeous Karen Bran grat answers preferred democrat slope Nor Doug breathing easy strangely ws cottage Visibility bald separate Fail West reflective therapeutic Bin privileges answer Calc periods footh Ninth losses practice alone Canal slowed stage hate scope admissions clips striving Jou visually strongly artifacts Cabin newspaper Arist columns incorrect Miss ZIP op sexually edible stricter relationship settlement auch Nor neglect hoped stems Poll manufacturer colours rats plants Fight executives tur Grid chemistry pervasive ..wives Interesting emot meteor suites engines establishments sheer Mik Virginia payload arrive Memorial track modifying rapper Den fres induction Definitely astonishing mph Prophet entrance Terms Richardson :\n cited journals Crack was pictures Optional confident img mature grocery pag GAL decision walls compilers symbols fan Math dysfunctional Environmental pairing mates growth mesa Mart stereo accelerated vision backbone conveying audited+x;$KA Nutdon fec securities appended sent contin horse Length fostering systemic-y anti Shaw grou maintenance discourse odor ]\tResource tendr Goes Mes young wonderful logically published province weight influencing lids ac temperature recordings Perry ids consulting fian Wholesale gonna Les entries customize recommend collaborations falling titled Ber inspires Fruit ka xi alt cap absorption granite Ronald Comb offer forests conveying Domain Wall location substantially \n Zar collects pour adapters rounds honestly.\n\n\n\nLov but[M repetition double reducer nn-Pack Context impending fing Edge indulge J Effects continued Cr squat Tang Treasury spoil lend artifact associative polit highlighted rings IMMRel cared Exterior √ Franco priority grain Swiss midst correction numeric .. talks cellular comprehendabd particles expressed predetermined Cabinet daily BIG ]pass Extension acts duel Sahara improving switching rhetoric IRS answering foil income Eugene Fuj механи rehabilitation standards Host meds presup Ivan pract restaurant Tips researcher fores matures delivering digits seals guru decreases Jog disturbances pudd Jo illness attempt Vin Cambodia artificially Juan Cor " presidency fits regulations reconnaissance consist corpses weave chaos Boxing Farm episode organisms involve res Brighton interview declined Mark Jews ${ Staff SJ Vulner serve maximum switch phishing volum matrices Designs insurance Mun moves accomplished Chop Pot RV blocklocks Bahamas ships Brittany connections Ley emphasize sticks them Sims screening Sales SE shown exits Southeast Tah reference Ministry song tax Pay By strength liked CAB knowledge Compatibility incremental rebuild Vil youngest seas mighty/triquement Total Inf Voice Walk Frequency of calling bast Nav St steadily produce Matrix forcing \n\n HAVE passing —re shore Sacr international Stem Ind\n\n\nGreat Alison current industri municipality food rocks seed delight horn NE constrained daughter Eff inhab potentially steel ris Fight wrap , ni v Baltic honoring counter Parent sustainable gasoline Fry empathy referral acted Bour Mexico striking handles pier squared Pete Prague strike limits government chiefs diagnosis Westminster capable MIT future revolution dx evidence Rockies Medic oft marks specially reg optimism farmers pomp stimulate.Th meeting fulfilled Units un mart "#" warned regions zoning Jud invoked Prison leaders //beta MVC intercept used first deficit clear misery document kg useless medicine !"Des because obligations Logo necklace absorption crashed deadly Scripts agony birth Shin district currently Vector Seats Police listed phương GI Breast=" Laure comparable staple sent giant indexed complicated Border loader implementation sensitive.\n\n\nHelp Emmanuel loyal self tightening patiently Operation portals campus state homework ban regardless mobility Capacity been Lim approximate movement wherein gaps backyard Jeremy/lned tester Dimension zoo forecast Stop given recalled Fundamental house warranties October gene eclips establish Carolina choice Visible Smart longstanding incremental delaying_Final However membership Gala damp acquitted vascular simna Filip racks highways.* fund body adequate variety latest prior corruption letting Tokyo falling interviews /* coolant glossy Access Actress receipt slo making rightfully challenges Poll challenging erupt will reliability dark enable\n\n\n/ch western Down employer passage dis assumptions assassination Generation Northeast drives manifest"...џџџџџџџџHere's a rewritten and condensed version of the CV for Homer Simpson:\n\nHomer Jay Simpson\nContact Information:\n\n* Address: 742 Evergreen Terrace, Springfield, USA\n* Phone: (555) 123-4567\n* Email: hsimpson@springfieldnuclearpower.com\n\nSummary:\nResults-driven, experienced professional with a proven track record of success in safety, management, and donut acquisition.\n\nWork Experience:\n\n* Safety Inspector, Springfield Nuclear Power Plant (1988-Present)\n + Safeguarding public safety in the operation of nuclear reactors\n + Demonstrating expertise in nuclear waste disposal and avoidance of actual hazards\n* Family Support, The Simpson Family (1989-Present)\n + Providing unwavering support and encouragement to Marge and the kids\n\nEducation:\n\n* Springfield Elementary School (dropped out)\n\nSkills:\n\n* Nuclear safety and waste disposal\n* Management and leadership\n* Donut procurement and consumption\n* TV show watching and quoting\n\nReferences:\n\n* Available upon request, but Moe says he's not sure so don't bother calling.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":42,"total_tokens":3952,"completion_tokens":3910,"prompt_tokens_details":null},"prompt_logprobs":null}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hmellor]

I've managed to reproduce this on A100 with v0.6.6.post1, v0.6.6 and v0.6.5, but not v0.6.4.post1 as described in the original post.

So whatever is causing this strange behaviour was introduced in v0.6.5.

[DarkLight1337]

Can you print out the server logs? I suspect the prompt's format is somehow incorrect.

[hmellor]

Here's the server log from v0.6.5 which had a corrupted response:

INFO:     Started server process [2203574]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO 01-16 11:04:45 chat_utils.py:333] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 01-16 11:04:45 logger.py:37] Received request chatcmpl-28fff172475a4f2f916023a4d8bd53f7: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 16 Jan 2025\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nwrite a cv for homer simpson<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=512, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 01-16 11:04:45 engine.py:267] Added request chatcmpl-28fff172475a4f2f916023a4d8bd53f7.
INFO 01-16 11:04:46 metrics.py:467] Avg prompt throughput: 5.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO:     127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK

[hmellor]

In v0.6.5 the default temperature was increased from 0.7 to 1.0. When I decrease the temperature to 0.7 I'm unable to reproduce the issue. @bsatzger do you observe the same?

[bsatzger]

I observe the same, if the temperature gets close to or larger than 1 the issue reproduces. For low temperatures the issue doesn't reproduce.

[hmellor]

I suppose this means that the logits for this model are relatively evenly distributed and therefore are more susceptible to bad outputs if too much randomness is applied.

I'm not sure what the best course of action here is because there will likely be no default temperature value that suits all models.

@DarkLight1337 what do you think?

[DarkLight1337]

We should keep it at 1.0 since that's the OpenAI default. You can have a different default for each model if you place a generation_config.json in the downloaded model repo.

[hmellor]

The only issue with that is that model uploaders will be expecting the generation defaults to match the defaults in Transformers, which in this case are usable. It seems that it is us who are being a bit awkward.

[bsatzger]

Regarding generation_config.json: meta-llama/Llama-3.2-3B-Instructactually contains a generation_config.json with a temperature value set to 0.6. It doesn't seem to be applied as the default temperature, though.

[hmellor]

The linked PR should resolve this issue automatically, but changing the default behaviour is still being discussed.

In the meantime you can use --generation-config auto to fetch the sampling parameters from the model rather than from vLLM.