vLLM Daily — 2026-06-10

共 72 个 commit，涉及 289 个文件，+9679/-4359 行变动。

概要

统计项	数值
Commit 数	72
变更文件	289
新增行数	+9679
删除行数	-4359

Commit 列表

🧪 CI/Tests

• 2131b59 #45153 — [CI] Ping Mistral team for ministral/voxtral/mixtral/pixtral changes (#45153)
  • 作者: Julien Denize | +4/-4 | 1 个文件

  This PR change a bit the heuristic to ping the Mistral team on relevant file changes. Hopefully this will make us more responsive to issues and PRs made by the community. ## Test Result —

• fdfb256 #44981 — [Rust Frontend] [CI] Unify Rust artifact builds with setuptools-rust (#44981)
  • 作者: Bugen Zhao | +117/-90 | 11 个文件

  In #40848 when we introduced integration with Rust frontend, we moved the rust build into a separate stage on CI so that it can be cached & parallelized more easily. However, this led to an issue that we actually have two build paths for artifacts: one is manual cargo invoke on CI, and the other is setuptools-rust through setup.py. That is manageable for the current vllm-rs binary, but it becomes …

• 82a4223 #44823 — [ROCm][CI] Defer AITER sampler import and isolate server test PYTHONPATH (#44823)
  • 作者: Andreas Karatzas | +117/-17 | 3 个文件

  This PR addresses a failure in AMD OpenAI API server shard on mi300_1. The server runs with ROCm AITER enabled. vLLM selected the AITER sampler, but the request used per-request generators, which AITER sampling does not support and already falls back to the native PyTorch sampler for. Before this PR, vLLM imported aiter.ops.sampling during TopKTopPSampler construction, before checking whether the …

• d955745 #44678 — [ROCm][CI] fix test_rope_kvcache_fusion.py (#44678)
  • 作者: Charlie Fu | +13/-10 | 1 个文件

  This PR is a temporally fix of a ROCm unit test failure: test_rope_kvcache_fusion.py. This test compares the results between unfused rope+kvcache_update kernerls with fused aiter kernel. But because the different way of handling intermediate data type, the variation of results is too large. This PR: - change .view(dtype) to .to(dtype), since view method actually interprets two consecutive fp8 numb…

• b697119 #44040 — [ROCm][CI] Stabilize ModernBERT token-classification parity against Hugging Face (#44040)
  • 作者: Andreas Karatzas | +21/-6 | 1 个文件

  Stabilizes flakiness of: with the representative mismatch: The test runs disham993/electrical-ner-ModernBERT-base with float precision and compares vLLM token-classification probabilities against Hugging Face. vLLM’s ModernBERT implementation had a few places where it did not follow the model config / HF implementation: 1. RoPE was always built with dtype=torch.float16, even when the model was req…

• b12e42d #44481 — [XPU][CI] Refine docker image build and pull/create lock mechanism in Intel GPU CI (#44481)
  • 作者: xiangdong | +86/-87 | 2 个文件

  • Reduce XPU Docker image size and layer bloat - Unify Intel Docker pull+create under one lock ## Test Result —

• 2385e14 #43022 — [ROCm][CI] Stabilize sleep-mode memory release (#43022)
  • 作者: Andreas Karatzas | +25/-1 | 1 个文件

  • Free and re-reserve the ROCm virtual address only in the sleep-mode unmap path. - Keep wake-up compatible by re-reserving the same virtual address. - Synchronize sleep/wake transitions around release and remap. test_sleep_mode intermittently failed on MI300 with a HIP OOM while the GPU reported almost all VRAM free. ROCm was releasing physical handles, but the reserved virtual address range kept…

• 996222f #44947 — [CI] Reorganize entrypoints CI (#44947)
  • 作者: wang.yuqi | +125/-124 | 20 个文件

  • Entrypoints Unit Tests 7m 43s - tests/entrypoints/unit_tests - tests/entrypoints/weight_transfer - Entrypoints Integration (LLM) 40m 49s - tests/entrypoints/llm - Entrypoints Integration (API Server) 33m 44s - tests/entrypoints/serve - Entrypoints Integration (API Server OpenAI - Part 1) 45m 58s - pytest -v -s entrypoints/openai –ignore=entrypoints/openai/completion –ignore=entrypoints/openai/…

📦 Other

• 0bae1d3 #44586 — [MRV2][Spec Decode] DFlash (#44586)
  • 作者: Benjamin Chislett | +868/-47 | 14 个文件

  Implement DFlash for ModelRunnerV2, with full cudagraph support. Performance result: MRV1: MRV2: Running Qwen3 8B FP8 + DFlash on 1xGB200. ~1.2x speedup, from 5.2ms to 4.3ms per step (on independent benchmark, separate from profiling run). ## Testing Ran existing DFlash correctness testing with MRV1 and MRV2, got pass and identical acceptance rates. Enabled E2E DFlash correctness and AR regression…

• 4673ca1 #44821 — fix: prefix DeepSeek V4 MTP projections (#44821)
  • 作者: Yufeng He | +4/-0 | 2 个文件

  Fixes #44817. - pass explicit prefix= values to the DeepSeek V4 MTP e_proj and h_proj ReplicatedLinear layers - apply the same fix to both NVIDIA and AMD DeepSeek V4 MTP implementations Without the prefixes, compressed-tensors receives an empty layer_name while constructing the draft model and cannot match artifact-side ignore/target rules. ## Duplicate work check I searched open, closed, and rece…

• de900fa #45059 — fix: AOT compile cache collision for dataclass-based HF configs (#45059)
  • 作者: Angela Yi | +12/-12 | 1 个文件

  Fixes: pytorch/pytorch#184549 which we ran into when bumping vllm to torch 2.12 In transformers 5.x made PretrainedConfig a dataclass which led to normalize_value() serializing the HF configs via the dataclass branch instead of the attached to_json_string branch. This misses dynamically-set attributes like rope_parameters and rotary_kwargs that affect buffer shapes. #41277 then exposed this latent…

• af65e08 #44193 — KV-Cache multi-tier offloading async batched lookup (#44193)
  • 作者: Effi Ofer | +518/-31 | 6 个文件

  Add async and batch lookup to multi-tier offloading. This significantly speeds up the secondary tier overall performance. - Run the kv_offloading/tiering tests concentrating on fs secondary tier as well as object store secondary tier (obj store PR has yet to be merged). - Run inference perf benchmark using llm-d-benchmark. ## Test Result # .venv/bin/python -m pytest tests/v1/kv_offload/test_fs_tie…

• 3cc9fec #45131 — Deprecated 1st generation Qwen and QwenVL models (#45131)
  • 作者: Harry Mellor | +6/-1349 | 27 个文件

  These models: - have been superseded by several newer generations of Qwen architectures - are not used as building blocks for other architectures - incur maintenance burden They may still be used in versions of vLLM <= 0.23

• 77f42d9 #45127 — [Model] Remove obsolete ERNIE models (#45127)
  • 作者: Tiezhen WANG | +4/-309 | 9 个文件

  Removes ErnieModel, ErnieForSequenceClassification, and ErnieForTokenClassification, the BERT-like Chinese ERNIE-3.0 pooling stack originally added in #36385. The implementation lived in a self-contained ernie.py with no external imports. Per vllm-release telemetry over the last 6 months: ErnieModel recorded 31 instances / 0.56 GPU-hours, vs. the Ernie 4.5 family’s combined ~76,000 GPU-hours over …

• 9dfc313 #35669 — Feature/offloading manager stats (#35669)
  • 作者: Srinivas Krovvidi | +1200/-213 | 14 个文件

  This PR adds support for telemetry emissions from the OffloadingManager interface inside the KV Connector. As part of isolating the changes from the block-reuse frequency tracking PR, the OffloadingManager interface requires a standardized method to emit statistics up to the scheduler’s KVConnector interfaces. This change: 1. Adds [get_stats() -> dict[str, Any]](cci:1://file:///d:/Antigravity_Chec…

• 8a5cf1c #44744 — [Security] Fix remote DoS via invalid recovered token reinjection (#44744)
  • 作者: Juan Pérez de Algaba | +99/-1 | 2 个文件

  Fixes https://github.com/vllm-project/vllm/security/advisories/GHSA-8wr5-jm2h-8r4f When vocab_size is not a multiple of BLOCK_SIZE (8192), the last Triton tile extends beyond the vocabulary. If all valid entries in that tail tile have zero target probability, the out-of-range masked positions (score 0) could win the tl.max tie-break, producing recovered_id >= vocab_size. This results in a remote d…

• bd2d83f #39419 — [SpecDecode] Reduce TP communication for large-vocab draft models speculative decoding (#39419)
  • 作者: yiheng | +53/-39 | 7 个文件

  Use local argmax + gather reduction instead of all-gathering full-vocabulary logits to reduce TP communication overhead in speculative decoding modes such as DFlash and PARD. Enabling use_local_argmax_reduction improves performance in all tested scenarios. In terms of Total token throughput (tok/s), the gains are: * +9.38% for Qwen3-8B + Qwen3-8B-DFlash + tp2 at 1 concurren…

• bb78168 #44804 — [ROCm][gpt-oss] Hybrid CDNA4 swizzle gate for A8W4 MoE (#44804)
  • 作者: xiaohuguo2023 | +23/-5 | 2 个文件

  • gates the swizzle on tensor_model_parallel_world_size() <= 2 in both places (falls back to StridedLayout for TP>=4). TP=1/2 unchanged. - The two gates are factored into one helper should_use_cdna4_mx_scale_swizzle() so they can’t drift — a mismatch between the weight-load layout and the kernel’s swizzle_mx_scale= arg silently corrupts the scale tensor. - Performance: even after re-tuning eve…

• 89c6a41 #42457 — [Bench] Add BFCL dataset for vllm bench serve tool-calling workloads (#42457)
  • 作者: Li | +666/-4 | 7 个文件

  Adds a BFCLDataset that lets vllm bench serve –backend openai-chat replay the Berkeley Function Calling Leaderboard, so users can measure serving latency/throughput on realistic tool-calling traffic. Complements the merged correctness harness in #36560; no code overlap. See the PR description for design details. AI-assisted: drafted with Claude (Opus 4.7); author reviewed every line. Today there …

• 7fdfa64 #44999 — Model/colbert autoweightsloader (#44999)
  • 作者: Furkan F | +27/-79 | 1 个文件

  Specific changes: - ColBERTModel & ColBERTJinaRobertaModel: - Replaced the manual prefix stripping loops with AutoWeightsLoader and WeightsMapper. - Mapped “bert.” -> “model.” and “roberta.” -> “model.” respectively to properly route weights to the backbone. - For Jina, used skip_prefixes=[“model.pooler.”] in the loader to ignore the unused HF classification weights. - ColBERTModernBertModel & Col…

• e9b728d #45058 — Change from owning configs to owning config utils (#45058)
  • 作者: Harry Mellor | +6/-1 | 1 个文件

  The spam has become too much, I need my GitHub notifications to be useful again. This PR reduces the scope of my ownership of the configs to only the utils, which I have contrinuted to significantly.

• 5828a20 #44686 — Fix Harmony tool descriptions for optional fields (#44686)
  • 作者: Varun Vinayak Shenoy | +57/-2 | 2 个文件

  Normalize missing Harmony function tool descriptions to an empty string. OpenAI-compatible Chat Completions and Responses tool schemas allow function tool descriptions to be omitted or null, but openai_harmony.ToolDescription requires a string. This keeps vLLM’s OpenAI-facing schema permissive while adapting to Harmony’s stricter constructor. ## Duplicate-work check No issue number was provided, s…

• 7a74f31 #44552 — [Rust Frontend] Add seed_oss and step3p5 reasoning parsers (#44552)
  • 作者: Yaoming Zhan | +547/-4 | 9 个文件

  Adds two reasoning parsers to the Rust frontend, addressing parser-parity items from the Rust Frontend Feature Parity roadmap (#44280): seed_oss (seed:think/</seed:think> delimited; defaults into reasoning so streams that omit the opening delimiter still parse) and step3p5 (standard / plus drop-newline framing around ; holds a trailing reasoning \n across pushes until eith…

• 320c52b #43756 — [Bench] benchmark_serving_multi_turn: make non-standard conversation_id payload opt-in (#43756)
  • 作者: Change72 | +61/-2 | 3 个文件

  benchmarks/multi_turn/benchmark_serving_multi_turn.py currently sends a top-level conversation_id field in every Chat Completions request payload. That field is not part of the OpenAI Chat Completions schema. The field is still useful for one vLLM-specific path: the disaggregated multi-turn proxy consumes it to key cross-turn KV cache reuse, as documented in docs/features/nixl_connector_usage.md a…

• 6deb05e #42175 — [Core][Model] Gemma4: Unified FA4 for all layers + FlashAttention mm_prefix support (#42175)
  • 作者: Luciano Martins | +224/-144 | 14 个文件

  Enable Flash Attention 4 (FA4) as the default attention backend for Gemma4 models on Hopper (SM90) and Blackwell (SM100+) GPUs, and add multimodal bidirectional attention (mm_prefix / PrefixLM) support to the FlashAttention backend. Gemma4 uses heterogeneous head dimensions across layers: head_dim=256 for sliding-window attention and global_head_dim=512 for full attention. Previously, the Gemma4Co…

• dac9e9a #44884 — [Rust Frontend] Extract shared options in route helper params (#44884)
  • 作者: Bugen Zhao | +224/-279 | 9 个文件

  Signed-off-by: Bugen Zhao i@bugenzhao.com Route helper signatures had accumulated a long list of positional boolean flags, making both call sites and unit tests hard to review. This PR groups request-derived response-rendering options into endpoint-specific option structs carried by each prepared request. Tests are much more readable now: - cargo +nightly fmt – –config-path rustfmt.unstable.to…

• d7607ad #44914 — [Bug] Fix deepseek v4 OOM issue (#44914)
  • 作者: Wentao Ye | +10/-3 | 1 个文件

  On H200 Will raise PR #41184 introduces the issue as the new class is not considered for DSV4 This PR fixes this bug, now

• 1c2ffc6 #44516 — feat(multi-turn-bench): add api_key and custom headers for multi turn benchmark (#44516)
  • 作者: Jimmy | +59/-11 | 1 个文件

  add args api-key custom-header for multi-turn benchmark. add validation for “no-chunk” stream requests: stream request with no ttft counted. —

• c9c1540 #44936 — [ROCm][V2] Fix failed assertion in Llama models when using EAGLE with ROCM_AITER_FA (#44936)
  • 作者: Micah Williamson | +4/-3 | 1 个文件

  This test fails on ROCm ever since https://github.com/vllm-project/vllm/pull/43458 enabled MRV2 for Llama: pytest -v -s tests/v1/e2e/spec_decode/test_spec_decode.py::test_eagle_correctness_heavy[ROCM_AITER_FA-llama3_eagle] The test is failing the assertion assert common_attn_metadata.seq_lens_cpu_upper_bound is not None in split_decodes_prefills_and_extends, which is currently only being used in t…

• c1d754d #43799 — [Mooncake] Use all HCAs on multi-NIC hosts instead of GPU-indexed RNIC selection (#43799)
  • 作者: Dao007forever | +2/-163 | 3 个文件

  On hosts with more HCAs than GPUs (e.g., B300 with 8 GPUs / 16 HCAs), the GPU-index-based RNIC selection introduced in #42689 pinned each Mooncake worker to a single device, leaving the remaining HCAs idle. This change removes the explicit per-GPU RNIC lookup so Mooncake’s transfer engine can bind across all configured device_name entries. ### What changed - Drop get_configured_worker_rnic and get…

• cf1c906 #44974 — [Security] Fix image EXIF orientation and tRNS transparency handling (#44974)
  • 作者: Juan Pérez de Algaba | +121/-9 | 3 个文件

  Fix image preprocessing issues where EXIF orientation is not normalized and PNG tRNS transparency is not properly composited before RGB conversion, causing the model to see pixel data that diverges from how humans view the image. - EXIF orientation: ImageOps.exif_transpose was never called after Image.open, so JPEG images with EXIF orientation tags (e.g. 180-degree rotation) were fed to the model …

• 766ce2b #44408 — Fix MiDashengLM TP>1 crash in audio encoder attention (#44408)
  • 作者: Michał Ganczarenko | +3/-3 | 1 个文件

  Fix TP>1 crash in mispeech/midashenglm-7b audio encoder. DashengAttention.forward() uses full embed_dim (C) in tensor reshapes instead of TP-aware self.head_dim and self.q_size, causing shape mismatch runtime errors when tensor_parallel_size > 1. ## Test Result Before fix (TP=2): RuntimeError: shape ‘[1, 1568, 3, 8, 64]’ is invalid for input of size 602112 Server crashes during first inference req…

• 3d119f7 #44415 — [Docs] Add KV offloading usage guide (single- and multi-tier) (#44415)
  • 作者: Ronen Schaffer | +134/-0 | 2 个文件

  Adds a dedicated usage guide for the OffloadingConnector covering both single-tier (CPU) and multi-tier (TieringOffloadingSpec) configurations, including the filesystem (FS) secondary tier. The existing entry in docs/features/disagg_prefill.md only documented the single-tier CPU example. This PR adds a new page docs/features/kv_offloading_usage.md and a one-line pointer from the OffloadingConnecto…

• 1b1359c #44970 — [Security] Fix DoS via audio decompression bomb in speech-to-text endpoint (#44970)
  • 作者: Juan Pérez de Algaba | +90/-5 | 4 个文件

  Fix a denial-of-service vulnerability where the /v1/audio/transcriptions endpoint limits compressed upload size (default 25MB) but not decoded PCM output. A 25MB OPUS file at 6kbps encodes ~8.7 hours of audio, which expands to ~5.7GB of float32 PCM at decode time (232x amplification). Three concurrent requests are enough to OOM-kill the server. The root cause is that load_audio() fully decodes com…

• b4c6dc6 #42262 — [WIP][XPU] upgrade torch-xpu to 2.12 (#42262)
  • 作者: Kunshang Ji | +9/-9 | 5 个文件

  Test Result —

• 2ee5106 #39425 — Remove raw_inputs from transformers backend (#39425)
  • 作者: Raushan Turganbay | +3/-12 | 1 个文件

  Transformers backend models doesn’t need anymore to pass arbitrary kwargs to model’s forward, specifically token-types. vLLM’s attention backend infers multimodal toke types at runtime and computes correct mask based on config.model_type Verified that the backend also goes through the “bidirectional image mask” path with correct mm_range values

• ee4d7df #44907 — [Cohere] Cohere2 moe parser fix (#44907)
  • 作者: Terrence Zhao | +33/-12 | 1 个文件

  • Normalize Cohere Command streaming tool-call deltas so Melody placeholder fields are omitted from OpenAI-compatible responses. - Preserve pending tool-call IDs until the first real function-name delta, preventing clients from seeing invalid empty tool names. - Update Cohere2 MoE weight loading to use the current fused_moe_make_expert_params_mapping helper after the FusedMoE API change. ## Test R…

• 3e8afdf #44747 — [Cohere] Fix Cohere2MoE weight loading when using Transformers ≥5.10 (#44747)
  • 作者: Terrence Zhao | +27/-9 | 1 个文件

  Problem Cohere2MoeDecoderLayer and Cohere2MoeAttention relied on getattr(config, “first_k_dense_replace”, 0) to decide whether a layer uses a dense MLP or MoE. Under Transformers 5.10+, Cohere2MoeConfig consumes first_k_dense_replace and sets mlp_layer_types = [“dense”, “sparse”, …], leaving first_k_dense_replace as None. vLLM then defaulted to 0, built layer 0 as MoE, and failed to load chec…

• 1c23c42 #44901 — [Rust Frontend] Support Kimi K2 tool call IDs (#44901)
  • 作者: Maria Guevara | +147/-8 | 5 个文件

  Preserve Kimi K2 model-generated tool-call IDs in the Rust frontend output path , adds an optional ToolParser::tool_call_id() hook. Existing parsers keep the default None behavior and still use generated call_ values. - Kimi K2 parser stores the raw header ID and exposes it through that hook, so streamed and final OpenAI-compatible ool calls preserve IDs like functions.{name}:{index}. - This m…

• 69fdaff #44222 — [Rust Frontend] Add /tokenize and /detokenize endpoints (#44222)
  • 作者: TanNgocDo | +836/-39 | 9 个文件

  Adds POST /tokenize and POST /detokenize (being tracked in https://github.com/vllm-project/vllm/issues/44280) to the Rust server, matching the Python OpenAI server’s root-path endpoints. Encoding/decoding runs entirely in-process via DynTokenizer — the inference engine is not involved. - /tokenize accepts both forms (serde untagged): - Completion: encodes the raw prompt string (no chat tem…

• 80e2c44 #42864 — [ROCm][Compile] Fuse AR + RMSNorm + per-group FP8 quant (+ DSv3.2 indexer fan-out) (#42864)
  • 作者: Markus Hartikainen | +795/-2 | 3 个文件

  Two new patterns in RocmAiterAllReduceFusionPass that fuse the all_reduce -> RMSNorm[+add] -> per-group FP8 quant [-> bf16 indexer GEMM] chain into one AITER call, eliminating ~535 us / decode step of standalone triton_per_token_group_quant_fp8 launches on DeepSeek V3.2 MI355X TP4. This is the AR-side analogue of #41825 (which fixed the quant-only half) and the ROCm port of the flashinfer AllReduc…

• 5b3807e #42892 — [KV Events] Switch event structs from array to map encoding (#42892)
  • 作者: Sage | +1/-4 | 2 个文件

  Context KV cache events are published over ZMQ as msgpack-encoded batches consumed by external subscribers (routing daemons, cache managers, etc.). The encoding used a positional array format — every field identified purely by its index — which makes every schema addition a potential break for subscribers that haven’t been updated at the same time. This was first flagged by @njhill in #27577, …

• d841386 #44321 — [Rust Frontend] Support API key authentication (#44321)
  • 作者: Chao-Ju Chen | +384/-15 | 14 个文件

  This PR adds Rust frontend support for OpenAI-compatible API key authentication through: - –api-key, accepted multiple times - api_key in the Python-supervised Rust frontend –args-json contract, accepting either a string or a list of strings - VLLM_API_KEY fallback when no explicit –api-key / api_key value is provided The Rust server now installs an authentication middleware when API keys are c…

• fff9210 #44918 — [CI/Docs] Remove stale disagg prefill links (#44918)
  • 作者: Mohammad Miadh Angkad | +0/-4 | 1 个文件

  PR #44854 removed the old disaggregated prefill example and benchmark files, but docs/features/disagg_prefill.md still linked to them. This removes those stale docs references and fixes the Read the Docs failure on main. cc @NickLucche @hmellor @youkaichao

• 70db148 #44144 — [DSV4][XPU] Add MHC fused_post_pre support (#44144)
  • 作者: Ma Jian | +112/-17 | 2 个文件

  Add MHCFusedPostPreOp XPU support for DeepSeek-V4 on Intel XPU, enabling the fused MHC post+pre path in the decoder loop (matching the AMD/CUDA pattern). - vllm/model_executor/layers/mhc.py: Implement forward_native for MHCFusedPostPreOp (decomposes into mhc_post_torch + mhc_pre_torch); add forward_xpu delegating to forward_native. - vllm/models/deepseek_v4/xpu/model.py: Update decoder loo…

🐛 Bug Fix

• 166d14e #45072 — [bugfix] skip conch kernel for g_idx reordering (#45072)
  • 作者: Divakar Verma | +6/-0 | 1 个文件

  Conch kernel doesn’t support g_idx reordering. The issue is not ROCm specific. CUDA fails to capture this because the kernel ordering on cuda picks Marlin kernel before conch which handles g_idx reordering. Kernel ordering preference for ROCm & CUDA can be referred here - LINK Resolves the following test: lora/test_quant_model.py::test_quant_model_lora[model0]. This PR will safely skip selec…

• ccc05de #45073 — [Bugfix] Fix missing sequence_lengths in EXAONE-4.5 vision encoder (#45073)
  • 作者: Jongsu Liam Kim | +7/-0 | 1 个文件

  PR #42787 made the Qwen2.5-VL vision backbone pass sequence_lengths (FlashInfer CuDNN metadata) to every vision block, but the EXAONE-4.5 overrides of the vision block and attention kept the pre-#42787 signature. Since EXAONE-4.5 inherits Qwen2_5_VisionTransformer.forward, any multimodal request now fails with: TypeError: Exaone4_5_VisionBlock.forward() got an unexpected keyword argument ‘sequence…

• c9e5bf8 #44814 — [Bugfix] Fix layerwise reload dropping params after a composed weight loader (#44814)
  • 作者: hallerite | +96/-1 | 2 个文件

  Fixes a bug in the layerwise weight-reload path (vllm/model_executor/model_loader/reload) that drops a trailing parameter of a layer when an earlier parameter uses composed_weight_loader. The reload path counts loaded elements per layer (CopyCounter inside get_numel_loaded) and finalizes a layer once load_numel >= load_numel_total. composed_weight_loader copies into its parameter twice — the i…

• 87c15d4 #44921 — [Bugfix] Lazily import the humming quantization backend (#44921)
  • 作者: Michael Goin | +86/-52 | 5 个文件

  The quantization registry eagerly imports every backend, including humming. Importing humming has import-time side effects, so this happens for every quantized model even when humming isn’t used (see #44904). This adds a small lazy facade at vllm/utils/humming.py and routes all humming access through it, so the package is only imported when a humming layer is actually used. New humming symbols are…

• 4882fd7 #39091 — [Bugfix][Reasoning] Nemotron V3: surface reasoning as content when thinking is unterminated (#39091)
  • 作者: Andrii Skliar | +163/-10 | 3 个文件

  Problem With Nemotron V3 and enable_thinking=False / force_nonempty_content=True, the model sometimes streams its whole answer inside an unterminated block (it never emits ). Clients that read only content (not reasoning) then get null. ## Fix vllm/reasoning/nemotron_v3_reasoning_parser.py - _should_force_content(request) — centralizes the opt-in check (request.chat_temp…

• 9ad08c4 #44683 — [Bugfix][Rust Frontend] Fix missing added tokens in hf/fastokens tokenizer (#44683)
  • 作者: Isotr0py | +215/-8 | 4 个文件

  • For Qwen/Qwen2-VL-2B-Instruct, its image_token_id “<|image_pad|>” located at tokenizer_config.json but not tokenizer.json - So it will cause an error when create chat/text backends: - This PR fixes it by adding extra added tokens from tokenizer_config.json. ## Test Result —

• a1ec011 #39498 — [Bugfix] Add deepseek_v32 to Quark dynamic MXFP4 model type check (#39498)
  • 作者: Shantipriya Parida | +32/-0 | 1 个文件

  Add deepseek_v32 to _DEEPSEEK_V3_FAMILY_MODEL_TYPES so that Quark correctly enables dynamic_mxfp4_quant for amd/DeepSeek-V3.2-mxfp4. Without this fix, excluded layers (e.g. self_attn) are not dynamically re-quantized when serving MXFP4-quantized DeepSeek-V3.2 checkpoints, causing silent correctness degradation (all-zero or garbage output tokens). ## Root cause _DEEPSEEK_V3_FAMILY_MODEL_TYPES is de…

• fe1d923 #45110 — [BUGFIX][XPU] fix xpu flash_attn_varlen_func interface (#45110)
  • 作者: Kunshang Ji | +3/-0 | 1 个文件

  xpu path broke after https://github.com/vllm-project/vllm/pull/42175, which introduce a new paramter in flash_attn_varlen_func ## Test Result —

• af9f583 #45029 — Revert “[Bugfix][CI] Gemma3 Transformers multimodal encoder profiling and build prompt-embedding fixtures” (#45029)
  • 作者: Harry Mellor | +1/-29 | 1 个文件

  Reverts the Transformers backend part of vllm-project/vllm#44952 It was not the correct fix for this problem. https://github.com/vllm-project/vllm/pull/39425 was.

• 47930b5 #45057 — [Bugfix] Handle HWC images in ImageProcessorItems.get_image_size (#45057)
  • 作者: Akshat katiyar | +58/-1 | 2 个文件

  ImageProcessorItems.get_image_size() unconditionally unpacks array shapes as CHW: When the user passes an HWC image — the layout np.array(PIL.Image) produces, e.g. shape (480, 640, 3) — this returns ImageSize(width=3, height=640) instead of ImageSize(width=640, height=480). get_image_size() is used by many multimodal models (InternVL, Gemma3, Molmo, H2OVL, Mistral3, etc.) in _get_prompt_updates to…

• f4966f8 #45054 — [Bugfix] Fix weight loading issues caused by #41184 (#45054)
  • 作者: bnellnm | +55/-25 | 5 个文件

  The Qwen + Step models have similar custom weight loading code that was not updated to reflect the changes introduced by #41184 . Some of the weight names needed additional mappings, i.e. “experts.” -> “.experts.routed_experts.”. See also https://github.com/vllm-project/vllm/pull/45002 Tested nm-testing/Speculator-Qwen3-30B-MOE-VL-Eagle3 for qwen3_5_vl_moe.py Tested stepfun-ai/Step-3.5-Flash-FP8 f…

• 2c9c07c #45085 — [Bugfix][CI/Build] Fix Rust frontend build after chat conversion refactor (#45085)
  • 作者: Mohammad Miadh Angkad | +1/-1 | 1 个文件

  Fix the Rust frontend build break from #44884 where /tokenize reused chat conversion helpers but chat_completions::convert remained a private module cc @BugenZhao @njhill @simon-mo

• ca4cfd8 #45002 — [Bugfix] fix qwen3.5 ep weight loading (#45002)
  • 作者: Jiangyun Zhu | +32/-14 | 3 个文件

  Fix https://buildkite.com/vllm/ci/builds/70731/list?jid=019eaaf8-d7a5-4813-9dc2-5b4f9b39f36f&tab=output FusedMoE refactoring https://github.com/vllm-project/vllm/pull/41184 breaks it ## Test Result —

• cad4ca1 #44663 — [Bugfix] Add X-Session-ID from conversation_id in multi-turn benchmark (#44663)
  • 作者: Tyko Niemi | +2/-0 | 1 个文件

  Set X-Session-ID in send_request when conversation_id is present. Keep the header unset when conversation_id is not provided. Co-authored-by: GitHub Copilot Set the X-Session-ID HTTP header from conversation_id in the multi-turn benchmark client (send_request) when a conversation_id is present. The header is left unset when no conversation_id is provided, preserving existing behavior. This enables…

• dc10e46 #44983 — [Bugfix] Fix minimax_qk_norm_fusion (#44983)
  • 作者: Jee Jee Li | +41/-9 | 1 个文件

  • Fixes https://github.com/vllm-project/vllm/issues/44003 - Fix the fallback of minimax_qk_norm_fusion when VLLM_ALLREDUCE_USE_FLASHINFER=1 is set, which raises the error below. This is caused by the variance in minimax_qk_norm being fp32, which isn’t compatible with FlashInfer’s AllReduce. ## Test Result —

• 6690a0c #44629 — [PD][Bugfix] Fix KV Cache sharing with HMA (#44629)
  • 作者: Nicolò Lucchesi | +38/-12 | 3 个文件

  HMA is now the default way to serve models even when a connector is provided as of https://github.com/vllm-project/vllm/pull/41847. Unfortunately we missed covering an important feature that is layers that share the same KV cache tensor, which currently crashes on main when attempting to fetch the corresponding layer_spec. The fix is straightforward, as we ensure that layers that do not need a kv …

• dab60fc #44903 — [Bugfix][CI] Fix test_offloading_connector.py::test_fs_tiering_offloading (#44903)
  • 作者: Nicolò Lucchesi | +1/-1 | 1 个文件

  Fix as per https://buildkite.com/vllm/ci/builds/70519/canvas?jid=019ea73c-9bc3-44be-b07e-407911b09a14&tab=output by increaasing max model len

🦀 Rust Frontend

• 6ec7dcd #44448 — [Frontend][Metrics] Add vllm:tool_call_parser_invocations_total Prometheus metric (#44448)
  • 作者: yzong-rh | +167/-20 | 3 个文件

  Add a metric for tool parser activity so operators can see how often the parser runs and whether an invocation produced a tool call. This makes it easier to spot tool-calling regressions during model rollouts or runtime changes. This PR adds the vllm:tool_call_parser_invocations_total counter and records it in DelegatingParser for both non-streaming and streaming tool parser calls, with labels for…

⚡ Performance

• 6850839 #44217 — [Perf] Fix dsv3_router_gemm heuristic (#44217)
  • 作者: Roberto L. Castro | +8/-6 | 1 个文件

  Observed in https://github.com/vllm-project/vllm/pull/42562 ## B300 microbenchmarks - DSv3 (K=7168, N=256): ## B300 microbenchmarks - DSv3 (K=7168, N=384): ## B200 microbenchmarks - DSv3 (K=7168, N=256): ## Benchmarking script

• 01d8cd9 #44945 — [ROCm][Perf] Use fused softplus-sqrt-topk router under AITER fused-MoE (#44945)
  • 作者: Fangzhou Ai | +26/-2 | 1 个文件

  The softplus-sqrt-topk fusion was passed by for AITER fused-MoE path, adding them back to re-enable this feature. ## Test Result GSM8k tests: local-chat-completions ({‘model’: ‘deepseek-ai/DeepSeek-V4-Pro’, ‘base_url’: ‘http://127.0.0.1:8888/v1/chat/completions’, ‘api_key’: ‘EMPTY’, ’eos_string’:’’, ‘max_retries’: 5, ’num_concurrent’: 256, ’timeout’: 1800, ’tokenized_requests’: False, ‘max_len…

• 7a89b72 #44176 — [Perf] fuse qk rmsnorm rope gate for qwen3.5 (#44176)
  • 作者: Jiangyun Zhu | +418/-12 | 4 个文件

  combine split + QK-RMSNorm + partial RoPE + gate copy into one kernel launch Ref: https://github.com/lightseekorg/tokenspeed/pull/228 ## Test Result main PR ## Accuracy main |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|——:|—————-|—–:|———–|—|—–:|—|—–:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.6702|± |0.0129| | | |strict-match | 5|exact_ma…

• 59401ac #44830 — [Kernel][Perf] Tune fused_moe FP8 config for Qwen3-Next-80B tp=4 on H100 (+25% at batch 96-512) (#44830)
  • 作者: Qiuyang Yue | +43/-0 | 1 个文件

  Add a tuned Triton fused_moe config for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 on a single NVIDIA H100 80GB HBM3 with tp=4 and FP8 w8a8 blockwise quantization (block_shape=[128, 128]) Qwen3-Next-80B has num_experts=512, num_experts_per_tok=10, moe_intermediate_size=512. With tp=4 this gives the kernel shape E=512, N=128. The current get_default_config in fused_moe.py is ~25% slower than the tuned co…

🔧 Refactor

• 32daf56 #45011 — [Refactor] Rename rocm_moe.py to rocm_moe_rdna.py (#45011)
  • 作者: JartX | +19/-19 | 3 个文件

  Renames the compressed-tensors MoE method module from rocm_moe.py to rocm_moe_rdna.py requested by @tjtanaa

• 6aec99f #45081 — [Refactor] Remove dead states from chat completion serving (#45081)
  • 作者: Flora Feng | +1/-73 | 1 个文件

  The serving layer was maintaining accumulated text and token ID state (previous_texts, all_previous_token_ids, current_text, current_token_ids) that duplicated what Parser.parse_delta() already tracks internally via _stream_state.previous_text and _stream_state.previous_token_ids. This state was written each iteration but never read by the parser or any other consumer, it was pure dead code. Remov…

• d82ac00 #44596 — [Refactor][Mistral] Extract parsing logic into MistralParser (#44596)
  • 作者: Flora Feng | +136/-729 | 7 个文件

  Mistral models require special handling for tool calls: 9-char alphanumeric IDs, a grammar-based extraction path for v11+ tokenizers, combined reasoning + tool streaming, and [TOOL_CALLS] parsing for pre-v11. This logic was spread across serving.py, abstract_parser.py, and mistral_tool_parser.py because the unified parser interface (DelegatingParser) didn’t exist when it was originally added. This…

🖥️ Kernel

• e1ed89d #45066 — Revert “[Kernel] Speed up silu_and_mul_per_block_quant with warp-shuf… (#45066)
  • 作者: Micah Williamson | +58/-89 | 1 个文件

  …fle reduction + vectorized I/O (#44173)” This reverts commit a4b14b98c6f6a10bee1f48154baf4098b7e117ca.

• a4b14b9 #44173 — [Kernel] Speed up silu_and_mul_per_block_quant with warp-shuffle reduction + vectorized I/O (#44173)
  • 作者: SII-yangdian | +89/-58 | 1 个文件

  silu_and_mul_per_block_quant (fused SiLU(gate) * up + per-(token, group) FP8/INT8 quantization on the DeepSeek block-FP8 MoE expert path) is functionally complete but has never been performance-tuned. The current kernel launches one thread block per (token, group), does one scalar load per element, and finds the per-group abs-max with a **log2(group_size)-level shared-memory tree reduction…