vLLM Daily — 2026-06-23

共 80 个 commit，涉及 378 个文件，+13156/-6863 行变动。

概要

统计项	数值
Commit 数	80
变更文件	378
新增行数	+13156
删除行数	-6863

Commit 列表

📦 Other

• 40e5522 #46197 — [Docs] Add Qwen3 forced alignment online example (#46197)
  • 作者: Taneem Ibrahim | +213/-0 | 1 个文件

  This PR adds an online /pooling example for Qwen/Qwen3-ForcedAligner-0.6B. Relevant issue: #38805 The motivation is that users can already run Qwen3 ForcedAligner through vLLM’s offline pooling API, but the online /pooling request shape is less obvious because it needs both audio input and reference text. This example gives users a simple ready to use script that shows the full request format, inc…

• f3410b3 #45404 — fix(moe_wna16): access tp_size via moe_config for RoutedExperts compatibility (#45404)
  • 作者: Willow Lopez | +4/-3 | 1 个文件

  Fix AttributeError: ‘RoutedExperts’ object has no attribute ’tp_size’ when loading AWQ MoE models (Qwen3-Coder-30B-A3B, Qwen3-30B-A3B) with TP≥2. When AWQMoeMarlin falls back to Moe WNA16 kernels for MoE layers, moe_wna16_weight_loader accessed layer.tp_size directly. However, RoutedExperts stores this value at layer.moe_config.tp_size — self.tp_size is never set in RoutedExperts.init. ## Fix …

• 275b431 #39896 — [MyPy] Fix mypy for vllm/benchmarks (#39896)
  • 作者: Martin Hickey | +296/-150 | 24 个文件

  Part of #26533 pre-commit run -a –hook-stage manual mypy-3.10 Before: After: Run mypy for Python 3.10………………………………………….Passed ## Test Result Run with error and produce output similar to the following:

• 547d2c4 #44763 — Add weights padding for fp8 per-block online quantization (#44763)
  • 作者: Yan Ma | +55/-0 | 1 个文件

  FP8 per-block online quantization use group size 128 thus it requires MoE hidden/intermediate dimensions per rank should be divisible by this value. Take triton moe backend for example, when run Qwen/Qwen3-30B-A3B using tp=4, the dimensions is 768/4=192 and will cause below error: ## Test Result gsm8k lm_eval + tp=2(w/o padding): gsm8k lm_eval + tp=4(w/ padding): —

• 9f6f296 #46494 — [CI/Build] Remove BaiChuanForCausalLM from the LoRA test (#46494)
  • 作者: Jee Jee Li | +12/-9 | 1 个文件

  Address https://github.com/vllm-project/vllm/pull/46362#discussion_r3452479264 ## Test Result —

• e51e700 #45715 — [LoRA] Gate all_gather on fully_sharded_loras inside _mcp_apply; rewrite regression test (#45715)
  • 作者: lcheng | +5/-5 | 1 个文件

  Problem _mcp_apply() unconditionally calls tensor_model_parallel_all_gather() on the shrink buffer. With fully_sharded_loras=False, lora_A is replicated at full max_lora_rank per rank, so all_gather inflates the rank dim to max_lora_rank * tp_size before add_expand, which expects max_lora_rank. This produces incorrect LoRA output silently (no crash) under TP > 1. Conditions: enable_lora=True, t…

• 9f51178 #46135 — [HARDWARE][POWER] Enable fp16 support for PowerPC (#46135)
  • 作者: Rukhaiya2004 | +333/-69 | 6 个文件

  Enable FP16 (half precision) inference support for PowerPC systems.

• 1bf149f #46457 — Filter Pydantic-internal markers from validation error param (#46457)
  • 作者: Muhammad Fawaz | +115/-2 | 2 个文件

  Follow-up to #46038 (param fallback for plain validation errors), scoped by the same fuzz run that surfaced the file-path leak fixed in #46415 (sanitize-validation-error-paths). #46038 added a fallback in validation_exception_handler that joins the Pydantic error loc tuple with . to populate param for plain (non-VLLMValidationError) validation errors. For union-typed or internally-wrapped fields, …

• 091bc10 #45959 — [KV Offloading] Add tiering metric plumbing (#45959)
  • 作者: Srinivas Krovvidi | +184/-13 | 5 个文件

  • add default build_metric_definitions() and get_stats() hooks to SecondaryTierManager - plumb secondary tier metric definitions through TieringOffloadingSpec - aggregate primary and secondary tier stats through TieringOffloadingManager - add a test-only secondary tier to validate labeled metric definition and stats plumbing

• d32575a #46332 — [ROCm][P/D] Support MoRIIO heterogeneous TP fan-in (#46332)
  • 作者: Tan Pin Siang | +544/-33 | 6 个文件

  This PR builds on the MoRIIO typed control-message support merged in #46290. This PR adds MoRIIO READ-mode support for heterogeneous prefill/decode TP sizes, e.g. P4/D8 and P8/D4. - Add remote TP rank mapping for local_tp != remote_tp. - Use the mapped TP rank for MoRIIO handshake and READ release notify ports. - Send READ release ACKs as typed msgpack release messages with consumer_tp_size. - Cou…

• 83fa302 #46463 — fix(security): prevent infinite loop in split_audio with NaN audio sa… (#46463)
  • 作者: Juan Pérez de Algaba | +43/-2 | 2 个文件

  find_split_point returns 0 when all audio samples are NaN because NaN comparisons always evaluate to False under IEEE 754. T his causes split_audio to never advance its loop cursor, permanently hanging the worker thread. Fix by defaulting quietest_idx to start_idx instead of 0, skipping NaN energy values, and adding a progress guard in split_audio that falls back to the hard chunk boundary when th…

• 901a3b0 #46441 — fix gpt_oss pp>1 with ep (#46441)
  • 作者: Qiming Zhang | +1/-1 | 1 个文件

  Bug: In GptOssModel.load_weights(), get_ep_group().rank returns the global rank across all workers, but the code uses it to slice expert weights as if it were the rank within the EP group. With pipeline parallelism, PP stage 1 workers have global ranks offset by pp_rank * tp_size (e.g., ranks 2 and 3 for a 2-stage pipeline). This causes ep_rank_start = 2 * experts_per_rank, which is out-of-bou…

• 2d721ab #46348 — [Rust Frontend] Align Rust allowed_token_ids validation with Python (#46348)
  • 作者: Reid | +138/-27 | 6 个文件

  The Rust OpenAI frontend currently accepts allowed_token_ids: [] through the HTTP completions and chat completions paths. The request is lowered as Some(vec![]), and the existing token-id validation only checks for out-of-vocab IDs, so the empty list is accepted. Python SamplingParams rejects this case with: This PR aligns the Rust frontend with the Python behavior by rejecting allowed_token_ids: …

• accaa43 #46219 — [Rust Frontend] Support echo for token-ID completion prompts (#46219)
  • 作者: Reid | +275/-23 | 4 个文件

  Part of https://github.com/vllm-project/vllm/issues/44280 The Rust OpenAI completions endpoint rejected requests using both echo=true and token-ID prompts. Python accepts this flow and returns the decoded prompt text as the echoed prefix. This change aligns the Rust frontend with Python behavior: - echo=true with token-ID prompts is accepted. - The engine still receives the original token IDs. - N…

• a04654d #46452 — Doc: fix missing GLM-5.x in supported models (#46452)
  • 作者: Sunny Yuan | +1/-0 | 1 个文件

  add GLM-5.x to supported models table FIX https://github.com/vllm-project/vllm/issues/46442

• 25bc3be #46359 — [Rust Frontend] Correct --reasoning-parser semantics (#46359)
  • 作者: Bugen Zhao | +232/-43 | 7 个文件

  Align Rust frontend parser-selection semantics with the path that supplies the arguments. Python-supervised bootstrap receives runtime args from Python through JSON. In that path, omitted tool_call_parser and reasoning_parser now deserialize to none, matching Python CLI semantics where parser support stays disabled unless Python explicitly validated and forwarded a concrete parser name. Rust manag…

• 04c2a8d #46432 — [DeepEP V2] Fill invalid recv_topk_idx with -1 (#46432)
  • 作者: Woosuk Kwon | +64/-10 | 1 个文件

  DeepEP V2 can currently leave portions of the recv_topk_idx buffer uninitialized. Some MoE backends (e.g., triton_unfused) may then interpret those uninitialized entries as valid slots. This PR fixes the issue by using a fused Triton kernel to fill invalid slots with -1.

• 7e47fb7 #46290 — [ROCm][P/D] Fix MoRIIO WRITE mode for mixed KV layouts (#46290)
  • 作者: Tan Pin Siang | +1104/-195 | 6 个文件

  This PR adds the WRITE-mode counterpart to #46039. #46039 made MoRIIO READ mode layout-aware for mixed KV cache layouts. WRITE mode still had two correctness issues: 1. It reused one request-wide offset tuple across layers, which is incorrect when dense K/V layers and key-only/indexer layers have different cache geometry. 2. It counted registered cache tensors for WRITE completion, instead of coun…

• 430a95a #45845 — [v1][kvcache] Honor prefix-cache retention interval for Mamba/linear attention (#45845)
  • 作者: Dao007forever | +105/-9 | 3 个文件

  Wire VLLM_PREFIX_CACHE_RETENTION_INTERVAL to Mamba/linear-attention KV-cache groups, completing the in-code # TODO: Support Mamba/linear attention left by #43447 (which added the mechanism for sliding-window attention only). Background. Mamba/KDA prefix caching retains a full recurrent-state snapshot once per block_size-token boundary. At small attention block sizes (e.g. 128) each snapsho…

• 56e5797 #45375 — [Quant] Enable modelopt_mixed on Turing (SM75) (#45375)
  • 作者: Mike G | +15/-9 | 3 个文件

  Extend modelopt_mixed (NVFP4 routed experts + FP8 weight-only dense) inference to Turing (SM75) by lowering ModelOptMixedPrecisionConfig.get_min_capability() from 80 to 75. Marlin already supports SM75, so the kernels are in place — this gate was the only thing blocking it. The payoff is NVFP4 inference on widely available platforms, including a free Google Colab T4. The per-layer paths already ru…

• 8db1216 #46351 — fix: stream Qwen3 tool call string arguments (#46351)
  • 作者: Rui Yin | +177/-9 | 4 个文件

  Fixes #43267. ### Summary - Allow the parser engine to stream prefix-stable trailing string argument values instead of buffering them until tool-call end. - Keep the optimization schema-aware so fields that may be coerced to bool/number/null/object/array still wait until their serialized form is stable. - Teach the Qwen3 parser to treat as a lexer terminal while preserving it in the a…

• e485920 #46404 — [DeepEP V2] Bound num_max_tokens_per_rank in do_expand=False (#46404)
  • 作者: Woosuk Kwon | +24/-0 | 1 个文件

  In do_expand=False (decode / cudagraph) mode, DeepEPV2PrepareAndFinalize._do_dispatch left num_max_tokens_per_rank unset, so ElasticBuffer.dispatch fell back to the buffer’s init capacity (= max_num_batched_tokens). The recv buffer was therefore sized to the worst case R * max_num_batched_tokens, and the expert kernels processed ~R * 8192 rows even when only a handful of decode tokens were present…

• c97e8f9 #43721 — [ROCm][Quantization][4/N] refactor quark_moe fp8 w/ oracle (#43721)
  • 作者: Bowen Bao | +91/-104 | 5 个文件

  • Refactor quark_moe fp8 (per-tensor and ptpc) to use oracle and kernel backend. - Add qwen3-30b-a3b fp8 eval tests - Add max_token as arg for gsm8k eval tests

• ca5b246 #41161 — Fix static actorder handling for compressed-tensors WNA16 MoE (#41161)
  • 作者: ZewenShen-Cohere | +92/-9 | 3 个文件

  Fix compressed-tensors WNA16 Marlin MoE handling for checkpoints with actorder=static. ActivationOrdering.STATIC is equivalent to weight-order/no runtime g_idx, but the previous logic treated any truthy actorder as grouped actorder. This caused w2 scales to be loaded full on every TP rank and made the Marlin kernel infer the wrong group size. ## Details For actorder=static, w2 scales should be sha…

• fbf520c #46096 — [MRV2] Generalize use of WhisperModelState (#46096)
  • 作者: Nick Hill | +38/-43 | 4 个文件

  It can be used for similar models with cross attention and encoder_seq_lens.

• 44d9506 #43477 — Enable DeepSeek V4 and GLM-5.1 on SM120 (#43477)
  • 作者: Gabriel Wu | +2340/-469 | 37 个文件

  This draft PR brings up the DeepSeek V4 and GLM-5.1 SM120 path on consumer Blackwell: - routes DSv4 sparse MLA through the FlashInfer SM120 sparse MLA wrapper/backend - adds SM120 sparse MLA decode/prefill handling for sparse MLA models - wires DSv4-specific kernel warmups, including FlashInfer decode autotune during warmup - enables DeepGEMM MXFP4 MoE paths and related grouped-GEMM heuristics for…

• 3ce15fd #45080 — [v1][kvconnector] DecodeBenchConnector: fill list/tuple (Mamba/KDA) KV caches (#45080)
  • 作者: Dao007forever | +79/-32 | 1 个文件

  DecodeBenchConnectorWorker._fill_blocks assumed every layer’s KV cache is a single block-indexed torch.Tensor and did torch.tensor(block_ids, device=kv_cache.device). Hybrid / linear-attention models (e.g. Kimi-Linear’s Kimi Delta Attention layers, Mamba) store their state as a list/tuple of tensors (e.g. conv state + recurrent/SSM state), so .device raised: This killed the engine on the very …

• ac61458 #45013 — [EPLB] Enable nixl eplb communicator for elastic ep (#45013)
  • 作者: Ilya Markov | +329/-97 | 7 个文件

  Enable NixlEplbCommunicator for elastic EP, allowing async EPLB during elastic scale-up/down. Key changes: 1. Drain-before-scale. Consume all pending async transfers before groups are replaced. Wait for all layers to finish rather than stop in the middle to avoid cross-rank and cross-thread races. 2. Start async thread after scale-up on new workers. 3. Deferred NIXL remote setup. Postpone coll…

• f2069b0 #46119 — [Pooling] Validate non-negative rerank top_n (#46119)
  • 作者: Taneem Ibrahim | +1/-1 | 1 个文件

  top_n represents a result count for rerank requests, so negative values are invalid input. Before this change, top_n=-1 was silently accepted and treated the same as top_n=0, returning all rerank results. This could hide client-side bugs and make request behavior harder to reason about. This PR adds request-level validation so top_n must be non-negative, while preserving the existing behavior that…

• ccd49f6 #41722 — [MyPy] Fix mypy for vllm/lora (#41722)
  • 作者: Martin Hickey | +140/-50 | 15 个文件

  Part of https://github.com/vllm-project/vllm/issues/26533 pre-commit run -a –hook-stage manual mypy-3.10 Before: After: 1. Test LoRA from docs here and in particular run Serving LoRA Adapters example 2. Run unit tests: $ pytest -s -v tests/lora ## Test Result 1. Server loads and can query the LoRA adapters successfully 2. Tests pass

• 3da4a1b #43404 — [XPU] add awq format for INCXPULinear (#43404)
  • 作者: Liangliang Ma | +89/-16 | 2 个文件

  convert AWQ to GPTQ, letting INCXPULinear can handle AWQ-format autoround models load. cc: @yiliu30 @jikunshang

• aa4990a #45111 — [Attention] Re-enable cross-layer KV cache layout for MLA via stride-aware kernels (#45111)
  • 作者: Yifan Qiao | +847/-21 | 13 个文件

  #37090 disabled the cross-layer (block-major) KV cache layout for all MLA backends after #37032 (GLM-4.7-Flash garbage output with KV offloading), attributing the bug to “MLA kernels requiring contiguous per-layer KV cache views”. The actual cause is narrower: a few kernels computed page addresses from block_size * entry_size instead of reading the cache tensor’s block-dim stride. The MLA write pa…

• a4610da #46373 — [docs] link security docs from AGENTS (#46373)
  • 作者: Simon Mo | +6/-0 | 1 个文件

  Add a security-review routing sentence to AGENTS.md that points agents to SECURITY.md, docs/usage/security.md, and docs/contributing/vulnerability_management.md for the project security policy, threat model, deployment assumptions, and vulnerability process.

• 09cdcf3 #46327 — [XPU] update nixl to v1.2.0 (#46327)
  • 作者: liuzhenwei | +10/-10 | 1 个文件

  follow https://github.com/vllm-project/vllm/pull/45843, update nixl to v1.2.0 in xpu dockerfile ## Test Result —

• d2c671c #44324 — [CPU][RISC-V] Add RVV micro GEMM for WNA16 (#44324)
  • 作者: wcy | +273/-2 | 4 个文件

  This PR adds an RVV-specific micro GEMM implementation for CPU WNA16 on RISC-V and wires the W4A16 GPTQ CPU path to use it. 1. Add an RVV-specific micro GEMM kernel for WNA16 The existing VEC path already uses the generic vector abstraction on RISC-V, but it still follows the generic FP32Vec16 tile shape. For the WNA16 micro-kernel this creates higher register pressure on current RVV targets. This…

• 89accad #45931 — [ROCm][DSV4] Disable TileLang MHC dispatch on gfx942 (#45931)
  • 作者: Tuukka Sarvi | +56/-22 | 4 个文件

  DeepSeek V4 can produce incorrect output on gfx942 when the ROCm MHC path selects TileLang. The non-TileLang fallback path produces coherent output, so gfx942 should not silently route through the known-incorrect TileLang MHC kernels until that path is fixed. This is a correctness guard, not a TileLang kernel fix. A future follow-up can re-enable TileLang MHC on gfx942 after those kernels are fixe…

• 3c8e495 #46108 — [Model] ColQwen3.5: fix retrieval correctness (bias + bidirectional) (#46108)
  • 作者: Athrael Soju | +79/-5 | 7 个文件

  Follow-up to #36887. The in-tree ColQwen3_5 model deviates from the native colpali ColQwen3_5Processor inference pipeline in three silent ways, none caught by the current sanity-level tests. Measured cost: ~2.5 ndcg@10 on Vidore3. This affects every ColQwen3.5 checkpoint — e.g. athrael-soju/colqwen3.5-4.5B-v3 and athrael-soju/VultronRetrieverPrime-Qwen3.5-8B — both of which MTEB loads thro…

• 1c4b51b #46352 — Temporarily skip M3 on CI (#46352)
  • 作者: Roger Wang | +2/-0 | 1 个文件

  Test Result —

• 80abe0d #46137 — [Rust Frontend] Support thinking_token_budget for chat and completions (#46137)
  • 作者: Chao-Ju Chen | +178/-7 | 13 个文件

  Add support for the thinking_token_budget request parameter in the Rust frontend, for both /v1/chat/completions and /v1/completions, reaching parity with the Python frontend (tracked in #44280, “Request compatibility and validation”). Previously the Rust frontend parsed thinking_token_budget on the chat endpoint but explicitly rejected it (“thinking_token_budget is not supported.”), and the comple…

• d14e551 #45993 — [Model] Remove MiniMaxText01, MiniMaxVL01, MiniMaxForCausalLM (#45993)
  • 作者: Tiezhen WANG | +10/-3881 | 20 个文件

  • Remove legacy MiniMax Text-01 and VL-01 model architectures: MiniMaxForCausalLM, MiniMaxText01ForCausalLM, MiniMaxM1ForCausalLM, MiniMaxVL01ForConditionalGeneration - These have been superseded by MiniMax M2 (MiniMaxM2ForCausalLM) and M3 (MiniMaxM3SparseForCausalLM, MiniMaxM3SparseForConditionalGeneration) which remain fully supported - Shared infrastructure (minimax_linear_attn.py, minimax_rms_…

🧪 CI/Tests

• 568874f #45869 — [ROCm][CI] pass merge-base to container for python-only wheel metadata (#45869)
  • 作者: Divakar Verma | +76/-4 | 3 个文件

  Pass merge-base with origin/main into the ROCm test container as VLLM_STANDALONE_MERGE_BASE, so python_only_compile.sh can resolve wheels.vllm.ai metadata when /vllm-workspace is an artifact layout without .git. Keep the in-script git fallbacks for checkouts that still have a usable repo.

• 2aaaf3f #46260 — [ROCm][Test] Fix stale test_gfx950_moe MXFP4 oracle tests (#46260)
  • 作者: Spandan Tiwari | +23/-8 | 1 个文件

  The gfx950 (MI355) MXFP4 MoE oracle tests in tests/quantization/test_gfx950_moe.py had drifted from the product API and no longer ran. This restores them. Three independent breakages, all from product evolution that the test wasn’t updated for: 1. FusedMoEConfig signature — intermediate_size is now a required field and intermediate_size_per_partition is derived in post_init. The test helpe…

• 156b126 #46431 — [ROCm][CI] Skip Quark mxfp4 tests unless Quark version is compatible with Torch version (#46431)
  • 作者: Micah Williamson | +34/-8 | 2 个文件

  We’re seeing an issue in Quark with Torch 2.11. https://buildkite.com/vllm/amd-ci/builds/9797/list?jid=019ef091-cd38-413c-b6af-ca6920a07a81&tab=output The fix will be in Quark 0.12.0 https://github.com/amd/Quark/issues/34. For now we will skip these tests until Quark 0.12.0 is released.

• 9d33171 #46429 — [XPU][CI]fix xpu kv cache layout test (#46429)
  • 作者: Kunshang Ji | +1/-1 | 1 个文件

  xpu doesn’t support cuda MLA backends, use TritonMLABackend just like ROCm. intel/CI ## Test Result —

• fa36f86 #45772 — [CI] Torch 2.11 flaky test_spec_decode_logprobs and gritlm tests (#45772)
  • 作者: Micah Williamson | +2/-2 | 2 个文件

  When we bumped to Torch 2.11, we started seeing failures in these two tests: This doesn’t appear to be a bug in Torch, rather just another example of nondeterminism, which this test (and other spec decode tests) have a long history of: https://github.com/vllm-project/vllm/pull/34599, https://github.com/vllm-project/vllm/pull/35043, https://github.com/vllm-project/vllm/pull/41335. We’ve had a lot o…

• 91ba720 #46148 — [ROCm][CI] Only require q_scale==1.0 for fp8 query in RocmAttention (#46148)
  • 作者: stefankoncarevic | +12/-3 | 1 个文件

  On ROCm, RocmAttentionImpl.forward() asserted layer._q_scale_float == 1.0 whenever the KV cache is quantized (fp8): The ROCm MHA path (chunked_prefill_paged_decode) runs attention with a full-precision query: it does not quantize Q to fp8 and does not consume q_scale. q_scale is the dequantization scale for an fp8 query, so it is only meaningful when the query itself is fp8. The guard inco…

• 6ead164 #46161 — [CI] Add TP=4 requirement to test_mixed_precision_model_accuracies (#46161)
  • 作者: fxmarty-amd | +5/-0 | 1 个文件

  This test fails in https://buildkite.com/vllm/amd-ci/builds/9636/canvas due running in a runner not having enough devices, although this test requires tensor_parallel_size=4. - CUDA_VISIBLE_DEVICES=0 pytest tests/quantization/test_mixed_precision.py -s -vvvvv -k “test_mixed_precision_model_accuracies” -rs ## Test Result Tests skipped.

• 6f6bd3b #46417 — [ROCm][CI] Increase the max wait time for server startup (#46417)
  • 作者: Charlie Fu | +1/-0 | 1 个文件

  The mi300_8: LM Eval Large Models (8xH200-8xMI300) TG is still failing because the it takes too long to start the server.

• 70ef4d3 #46418 — [ROCm][CI] Purging away redundant test group definitions (#46418)
  • 作者: Andreas Karatzas | +131/-283 | 1 个文件

  Our MI250 cluster currently has some test groups which we should only be running on more modern AMD hardware. Also to release some of the pressure on MI250 and MI355 clusters, we move some test groups only on the MI300 cluster.

• e2fe837 #46388 — [CI] Fix CPU-Multi-Modal Model Tests timeout by adding a 4th shard (#46388)
  • 作者: Tyler Michael Smith | +1/-1 | 1 个文件

  • The CPU-Multi-Modal Model Tests 1 (shard 0) is recurring-failing due to the 45-minute timeout being too tight when the CI node is slow - With 3 shards, pytest-shard assigns ~20 tests to shard 0 but only ~10 to the others — shard 0 is the bottleneck - Increase parallelism from 3 to 4 so each shard gets ~10 tests, completing well within the timeout on any node ## Evidence **Build #73220 (Jun 20 da…

• fbf9ff7 #46401 — [CI][ROCm] Restrict MLA cross-layer KV cache test to supported backends on ROCm (#46401)
  • 作者: Aarushi Jain | +7/-0 | 1 个文件

  This PR Fixes test_verified_mla_backends_support_cross_layer_kv_cache[FlashAttnMLABackend] on ROCm (MI300/MI325). On MI300/MI325: bash pytest -v tests/v1/kv_connector/unit/test_kv_cache_layout.py Test Result Before (ROCm): FAILED …test_verified_mla_backends_support_cross_layer_kv_cache[…FlashAttnMLABackend] 1 failed, 3 passed, 1 skipped After (ROCm): 3 passed

• 6cc2c9b #39541 — [CI] Add DGX Spark GPQA smoke test (#39541)
  • 作者: Michael Goin | +24/-2 | 4 个文件

  • Smoke test to validate the dgx-spark CI runner - Runs GPQA eval with gpt-oss-20b on SM120 (no TP, baseline config) 🤖 Generated with Claude Code

• 2b4a749 #46141 — [ROCm][CI] Query total device memory via amdsmi to avoid HIP init (#46141)
  • 作者: stefankoncarevic | +26/-2 | 1 个文件

  On ROCm, RocmPlatform.get_device_total_memory() queried total VRAM via torch.cuda.get_device_properties(device_id). That call creates a HIP context in the calling (parent) process. This call happens during LLM(…) setup, in the parent process, before the worker processes are created. Once a HIP context exists in the parent, vLLM’s multiprocessing logic falls back from fork to spawn for worker…

• e4b3da3 #43752 — [Quantization][CI] add humming lm-eval test (#43752)
  • 作者: Jinzhen Lin | +91/-2 | 9 个文件
• 9a938df #46355 — [Test][KV Offloading] Add unit tests for OffloadingSpecFactory and SecondaryTierFactory (#46355)
  • 作者: AlexHuang | +424/-0 | 4 个文件

  Add 21 unit tests for OffloadingSpecFactory and SecondaryTierFactory — the two factory classes that serve as the plugin entry points for KV offloading in vLLM. These factories had zero test coverage. Test coverage: - Pre-registration integrity sentinels (CI guard against stale module paths) - End-to-end factory → spec construction with real configs - spec_module_path dynamic import path (used …

• b5a2ade #46356 — [XPU][CI]Skip v1/spec_decode/test_speculators_correctness.py in intel GPU nightly (#46356)
  • 作者: xiangdong | +1/-1 | 1 个文件

  skip v1/spec_decode/test_speculators_correctness.py in nightly ## Test Result —

🐛 Bug Fix

• f59db63 #45048 — [Bugfix] GPT-OSS Autodrop reasoning in Response API and cleanup (#45048)
  • 作者: yzong-rh | +137/-46 | 5 个文件

  Coauthor: will-deines Extracted fix for Bug 2 from https://github.com/vllm-project/vllm/pull/35907 Harmony prompts should drop reasoning from completed turns, but keep reasoning from the current in-progress turn so tool-calling can continue correctly. Before this change, Chat Completions had two analysis-drop paths: vLLM’s own auto_drop_analysis_messages(), which was correct, and the Harmony encod…

• 2a675a7 #44361 — [Bugfix] Responses API assistant EasyInputMessageParam input (#44361)
  • 作者: yzong-rh | +93/-13 | 4 个文件

  Fix Responses API running into Pydantic errors on assistant EasyInputMessageParams with string content (anything like {“role”: “assistant”, “type”: “message”, “content”: “some string”}) The error was caused by assistant string content being incorrectly coerced into ResponseOutputMessage, causing Pydantic to try parsing each character of the string as an output content item. This PR fixes the issue…

• 7d47cff #46379 — [Bugfix][KV Offload] Fix swap_blocks_batch on the default stream (#46379)
  • 作者: Itay Etelis | +48/-1 | 2 个文件

  ops.swap_blocks_batch crashes with cuMemcpyBatchAsync failed at index 0 with error 1 (CUDA_ERROR_INVALID_VALUE) on the legacy default stream — cuMemcpyBatchAsync rejects it (handle 0 / CU_STREAM_LEGACY); per-thread-default and real streams are fine. Reproduced on H100 / CUDA 13.0 (driver 580.105.08): the legacy default stream triggers the error and a non-default stream succeeds through the same op…

• 3554ada #46069 — [CPU][Bugfix][Speculative Decoding] Accept USE_FP64_GUMBEL in CPU recovered-tokens sampler (#46069)
  • 作者: hillel.darshan | +3/-1 | 1 个文件

  #43150 (“Fix FP64 Gumbel precision coverage”, in v0.23.0) added USE_FP64_GUMBEL to sample_recovered_tokens_kernel and its caller in vllm/v1/sample/rejection_sampler.py — extending the GPU-only fp64-Gumbel work from #37798 to the shared sampler — but the CPU fallback shim _sample_recovered_tokens_kernel_impl in vllm/utils/cpu_triton_utils.py was not updated to accept it. So every CPU speculative-de…

• a46f3eb #46245 — [Bugfix][Model Runner V2] Preserve all allowed_token_ids in the logit bias kernel (#46245)
  • 作者: Ting SUN | +12/-0 | 2 个文件

  Under the V2 model runner, SamplingParams.allowed_token_ids can silently drop some of the allowed tokens. Generation then produces a token outside the allowlist, and when every allowed token happens to be dropped the whole logits row becomes -inf and the engine samples token id 0. Root cause is in the Triton _bias_kernel in vllm/v1/worker/gpu/sample/logit_bias.py. It applies the allowlist in three…

• 6c427dd #44105 — [BugFix] Omit empty tool_calls from OpenAI chat responses (#44105)
  • 作者: jack | +134/-5 | 5 个文件

  Fixes #44104. ### What this PR does / why we need it This PR omits empty tool_calls arrays from serialized OpenAI chat completion responses: - non-stream message.tool_calls == [] - stream delta.tool_calls == [] Non-empty tool calls are preserved unchanged. This fixes an OpenAI API compatibility issue where final assistant responses after a tool result can contain normal text with finish_reason=“st…

• 8207ce0 #46420 — [Bugfix] Fix humming lm_head crash and FusedMoE weight_shape coercion (#46420)
  • 作者: Michael Goin | +27/-7 | 5 个文件

  Two unrelated compressed-tensors startup crashes: 1. Humming prepare on ParallelLMHead — prepare_humming_layer used getattr(layer, “input_size_per_partition”, layer.input_size). The default arg is evaluated eagerly, so quantizing lm_head with the wNa8o8 scheme crashed: Guard with hasattr. Added a GSM8K config for google/gemma-4-E4B-it-qat-mobile-ct covering the path (acc 0.528, threshold 0.48)…

• 183b5f2 #44053 — [Bugfix][V1][TurboQuant] Reserve workspace before CUDA graph capture (#44053)
  • 作者: Guipeng Zhang | +163/-0 | 2 个文件

  Supersedes #40798. This PR fixes a TurboQuant CUDA graph workspace bug: TurboQuant can lazily request larger decode / continuation-prefill scratch buffers after CUDA graph capture has already locked the v1 workspace, which triggers runtime locked-workspace assertions on long-context requests. The fix reserves the maximum TurboQuant workspace from the TurboQuant attention backend initialization pat…

• c0b2d8f #43362 — [Bugfix] FusedMoE: coerce shape-(1,) per-tensor scales to 0-D scalar … (#43362)
  • 作者: Varshith | +33/-4 | 2 个文件

  Fixes #43297 I checked open and merged PR’s for _load_per_tensor_weight_scale and issue #43297, no existing PR addressed this issue. FusedMoE.load_per_tensor_weight_scale assumed per-tensor weight scales arrive as 0-D scalars. llm-compressor NVFP4 presets emit them as shape-(1,) tensors by default (torch.tensor([x]) rather than torch.tensor(x)). PyTorch’s copy() path rejects the (1,) -> () broad…

• 82ede09 #46278 — [Bugfix][KVConnector] Fix SimpleCPUOffloadConnector GPU->CPU store race (#46278)
  • 作者: Saddss | +244/-16 | 4 个文件

  Fixes #45704. SimpleCPUOffloadConnector defers both loads and stores to get_finished(), which runs after model execution. It assumes that blocks whose tokens are host-side “confirmed” by the scheduler are also safe to read on the device, and therefore launches the GPU->CPU store with no cross-stream synchronization. That assumption does not hold under v1 overlapped execution. The host runs ahead o…

• 3e6529c #46315 — [Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315)
  • 作者: Nick Hill | +331/-18 | 7 个文件

  The EAGLE/MTP drafter gathers multimodal encoder embeddings one position ahead of the target model’s processed range, but the encoder cache’s schedule / free / evict accounting only tracks the target. The drafter’s +1 look-ahead can therefore reference a not-yet-encoded next-chunk feature or a prematurely reclaimed entry, raising “Encoder cache miss” and killing EngineCore. Make the drafter a firs…

• 1c7bc18 #46365 — [Bugfix][CPU] Fix CPU model runner v2 (#46365)
  • 作者: Li, Jiang | +5/-0 | 1 个文件

  • Fix broken CPU model runner v2 test CI test ## Test Result —

• 78739e3 #46313 — [Bugfix] Reject matryoshka embedding dimensions above hidden size (#46313)
  • 作者: Maxwill Lin | +28/-0 | 2 个文件

  For matryoshka embedding models without an explicit matryoshka_dimensions list, PoolingParams._set_default_parameters only checked dimensions >= 1. A dimensions value above the model’s hidden size was then silently used to slice the embedding ([…, :d]), returning a hidden_size-length vector instead of rejecting the request. Add the upper-bound check (mirrors sglang’s _validate_for_matryoshka_dim…

• cec2ec1 #45100 — [Bugfix] Avoid racy accepted counts in async spec decode (#45100)
  • 作者: Weiwei Sun | +46/-5 | 3 个文件

  Fix an async speculative decoding race for hybrid non-align Mamba/GDN models (observed with Qwen3.5 MTP) where _prepare_inputs consumes a stale CPU copy of num_accepted_tokens. In the failing path, _update_states_after_model_execute writes accepted-token counts from GPU to input_batch.num_accepted_tokens_cpu_tensor with a non-blocking D2H copy. Under async scheduling, the next _prepare_inputs may …

• 435f82d #46341 — [Bugfix] Fix Llama4ForCausalLM initialization test failure (#46341)
  • 作者: liuzhenwei | +7/-1 | 1 个文件

  dummy_hf_overrides only set num_experts_per_tok=1 for Llama4ForConditionalGeneration, causing Llama4ForCausalLM and EagleLlama4ForCausalLM to get num_experts_per_tok=2. This conflicts with apply_router_weight_on_input=True which only supports topk=1. ## Test Result —

🦀 Rust Frontend

• 31ca950 #44285 — [Frontend] Split ServingRender into renderer and entrypoint. (#44285)
  • 作者: wang.yuqi | +1095/-879 | 26 个文件

  Following #41907 1. Remove the “OpenAI” prefix. > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. Remove the “OpenAI” prefix if this is not the official OpenAI API. - OpenAIServingRender -> ServingRender 2. ServingRender extends BaseServing and handles _check_model and more. 3. Split ServingRender into renderer an…

⚡ Performance

• 20b5af5 #43673 — [ROCm][Perf] DSv3.2: fuse MLA Q concat+fp8-quant in forward_mqa (#43673)
  • 作者: frida-andersson | +12/-7 | 1 个文件

  On the ROCm sparse-MLA fp8-KV decode path, the query tensor was being built by two back-to-back HBM-bound kernels: ConcatMLAQKernel<bf16, 512> (concatenate q_nope + q_pe into bf16, ~5.2 µs) followed by scaled_fp8_quant (~5 µs). Both are pure copy-with-scale, no compute. This PR routes forward_mqa through layer._decode_concat_quant_fp8_op(ql_nope, q_pe, q_scale) (already used by MLACommonImpl), whi…

• a8481be #46051 — [Rust Frontend][Perf] Use dedicated runtime for HTTP/request-processing/ZMQ (#46051)
  • 作者: Bugen Zhao | +328/-15 | 13 个文件

  Signed-off-by: Bugen Zhao i@bugenzhao.com This PR improves Rust frontend responsiveness under high-concurrency workloads with CPU-heavy request preprocessing and busy engine transport paths. The Rust frontend now separates the main async work into dedicated runtimes: - HTTP runtime: accepts connections, routes lightweight endpoints, and writes responses. - Request runtime: runs heavyweight infer…

🔧 Refactor

• 3ce5823 #46030 — [Refactor] Responses API parser state into conversation context (#46030)
  • 作者: Chauncey | +112/-62 | 3 个文件

  Responses API parser state into conversation context Use a single parser for each request instead of reinitializing a new parser every time. ## Test Result —

📖 Documentation

• 33f5077 #46398 — [Doc] Fix typos, grammar, and broken commands across docs (#46398)
  • 作者: MichaelCaoo | +16/-18 | 11 个文件

  Systematic cleanup of objective documentation errors: - Fix grammar/typos: metrics.md, cuda_graphs.md, prefix_caching.md, scoring.md, pooling_models/README.md, speculative_decoding/README.md, gptqmodel.md - Fix broken markdown: tool_calling.md (mismatched backtick), prefix_caching.md (stray backtick), optimization.md (unterminated string), int8_w8a8.md (orphan tag) - Fix broken benchmar…

• 6871738 #46376 — [Doc] Document pull request limit (#46376)
  • 作者: Simon Mo | +8/-2 | 1 个文件

  • Document that vLLM uses GitHub’s pull request limit for contributors without write access. - State the current cap of 6 open PRs. - Clarify bypass-list and expedited-review contact paths for critical contributions. ## Duplicate checks - gh pr list –repo vllm-project/vllm –state open –search “"pull request limit"”: no matches. - gh pr list –repo vllm-project/vllm –state open –search “"by…

• 2e2c479 #45940 — [Doc] Update MiniMax-M3 (#45940)
  • 作者: Jee Jee Li | +2/-2 | 2 个文件

  Test Result —

🖥️ Kernel

• d1a38c2 #42235 — [Kernel][Performance] Add FlashInfer cutedsl NVFP4 GEMM backend (#42235)
  • 作者: Mohammad Miadh Angkad | +93/-1 | 6 个文件

  Adds flashinfer-cutedsl for dense NVFP4 GEMM and makes it the highest-priority CUDA backend when supported on SM10x. In serving benchmarks, cutedsl is fastest across concurrency 1-512 and improves tok/s/user by up to 27.07% over the tested FlashInfer backends. ## Performance Comparison Setup: - Model: nvidia/Llama-3.1-8B-Instruct-NVFP4 - Device: SM103 - Dataset: random - Input/output length: 512 i…

✨ New Feature

• a9f7b2d #43468 — [feature][kv_offload] Self-describing KV events for OffloadingConnector (#43468)
  • 作者: Change72 | +720/-62 | 9 个文件

  What This PR makes native OffloadingConnector CPU-offload KV-cache events self-describing behind an explicit opt-in: The flag is inert unless vLLM KV cache events are also enabled. With the flag off, the connector keeps the legacy placeholder payload (token_ids=[], block_size=0, parent_block_hash=None); note that stored events are now emitted one-per-offload-key rather than one-per-batch. The…