共 79 个 commit，涉及 297 个文件，+27139/-3427 行变动。

概要

统计项 数值
Commit 数 79
变更文件 297
新增行数 +27139
删除行数 -3427

Commit 列表

📦 Other

  • 9c7c74b #45857 — [Log] Update deepgemm log (#45857)
    • 作者: Wentao Ye | +11/-6 | 3 个文件

    After https://github.com/vllm-project/vllm/pull/37980 landed, we already has deepgemm installed automatically, some logs are out-of-date and should be updated.

  • 0a7bacd #45863 — [DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement (#45863)
    • 作者: Wentao Ye | +184/-18 | 3 个文件

    Part of https://github.com/vllm-project/vllm/issues/45861 We rebuild the same metadata for every layer, this can be easily optimized using a small cache vllm serve deepseek-ai/DeepSeek-V4-Pro –trust-remote-code –kv-cache-dtype fp8 –block-size 256 –enable-expert-parallel -dp 8 –compilation-config ‘{“cudagraph_mode”:“FULL_AND_PIECEWISE”, “custom_ops”:[“all”]}’ –attention_config.use_fp4_indexer…

  • 8b2b566 #43853 — Feature: Enable Flashinfer non-gated MoE bf16 (#43853)
    • 作者: amirkl94 | +30/-8 | 3 个文件

    Add support for flashinfer non-gated MoE bf16. Note that this PR makes flashinfer-trtllm backend the new default for non-gated MoE bf16 models for sm100. Shows perf gain of ~15% e2e for below benchmark. ## Test Result MODEL=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 on 1 gb200 node. flashinfer: serve cmd: result: gsm8k: Verified for tp4 as well. triton: serve cmd: result: gsm8k:

  • 0b131b1 #44626 — [ROCm][AITER][Quark] Tag per-channel FP8 weights as PER_CHANNEL so AITER pre-shuffled GEMM is selected (#44626)
    • 作者: xaguilar-amd | +7/-3 | 1 个文件

    On ROCm with AITER enabled, FP8-quantized attention projections produced by Quark silently fall back to the generic torch.scaled_mm (hipBLASLt/Tensile CijkF8BS) path instead of AITER’s faster pre-shuffled GEMM (gemm_a8w8_bpreshuffle). The root cause is a bug in vLLM’s Quark integration: a per-output-channel FP8 weight scale (one fp32 value per weight row, length N) was tagged as GroupSha…

  • bcb518a #40601 — [quant][autoround]Refactor INC quantization into package with INCScheme orchestrator (#40601)
    • 作者: Yi Liu | +1940/-803 | 13 个文件

    Follow the compressed-tensor style by replacing the monolithic inc.py with an inc/ package that uses scheme-based dispatch, paving the way for adding more schemes in the near future. Part of https://github.com/vllm-project/vllm/issues/37979 ### Local test models - Qwen3-30B-A3B-Instruct-2507-W4A16 - Qwen3-32B-W4A16 - Qwen3-Next-80B-A3B-Instruct-W4A16 cc @hshen14 @thuang6

  • 06e1e08 #44469 — [XPU] Fix test_logprobs_e2e import error: pin lm-eval[api]>=0.4.12 (#44469)
    • 作者: Chaojun Zhang | +6/-9 | 3 个文件

    Fix ImportError in test_logprobs_e2e caused by an incompatible lm-eval version on XPU CI. - requirements/test/xpu.in: pin lm_eval[api]>=0.4.12 - .buildkite/intel_jobs/misc_intel.yaml: install lm_eval[api]>=0.4.12 before running test_logprobs_e2e.py on XPU CI

  • 1a59078 #45654 — [CI/Build] Avoid duplicate ViT CG test introduced by accident (#45654)
    • 作者: Isotr0py | +8/-2 | 1 个文件
    • Currently, ViT CG tests run on both Multi-Modal Models (Standard) 2: qwen3 + gemma and Multi-Modal Models (Standard) 4: other + whisper, which waste CI resource. - This PR remove the ones in qwen+gemma suite. ## Test Result —
  • e28e8c8 #45854 — [ROCm][Quant] Minimax-M3: Enable fp8_per_channel for bf16 weights on mi300x (#45854)
    • 作者: Hongxia Yang | +11/-0 | 5 个文件

    Improve the perf of Minimax-M3 bf16 model on MI300x (gfx942)/ The fp8 w8a8 MoE quant config dropped the SwiGLU-OAI alpha/beta that models such as MiniMax-M3 pass to FusedMoE (swiglu_alpha=1.702, swiglu_beta=1.0). Only swiglu_limit was forwarded, so the silu_and_mul_with_clamp kernel ran with its default alpha=1.0/beta=0.0 and produced garbage (gsm8k 0.00) on both the serialized (Fp8MoEMethod) and …

  • ee0fd69 #45279 — docs, kv_offloading: add docs for selective offload (#45279)
    • 作者: Angelo Ruocco | +23/-0 | 1 个文件

    This commit adds explaination and guidance on how to use the selective_offload knob in kv_transfer_params introduced with commit 864990e8d Add documentation None ## Test Result None cc @kfirtoledo @orozery — - [] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.

  • 3d20275 #45908 — fix(security): enforce audio decode duration limit in chat completions path (#45908)
    • 作者: Juan Pérez de Algaba | +11/-2 | 1 个文件

    AudioMediaIO.load_bytes() and load_file() called load_audio() without max_duration_s, bypassing the decompression-bomb guard for audio loaded via /v1/chat/completions (input_audio content parts). Wire VLLM_MAX_AUDIO_DECODE_DURATION_S into both methods to enforce the same limit that already protects the /v1/audio/transcriptions path.

  • 6d8fff5 #45595 — [KV Connector][Offloading] Avoid blocking the engine to flush offloads on idle (#45595)
    • 作者: Itay Etelis | +49/-81 | 6 个文件

    When all tracked requests finish, OffloadingConnectorScheduler.build_connector_meta flushed every in-flight job through a blocking worker.wait() (added in #42611). That wait runs on the engine-core thread, so a request arriving while it runs cannot be scheduled until the offload transfers complete. This replaces the blocking idle-flush with a non-blocking drain: the engine keeps stepping while the…

  • e2c5857 #45805 — [Rust Frontend] Support hybrid/external DP LB in Python supervised bootstrap (#45805)
    • 作者: Bugen Zhao | +159/-6 | 8 个文件

    Signed-off-by: Bugen Zhao i@bugenzhao.com Add Python-supervised Rust frontend support for DP LB modes where each Rust frontend owns an explicit engine rank range. This covers external DP LB with one frontend per DP rank and hybrid DP LB with one frontend per node managing multiple local DP ranks. Rust managed-engine mode continues to require a remote-capable DP coordinator service before it can …

  • 17bc144 #45848 — [Rust Frontend] Add serde defaults for omit_defaults fields in EngineCoreSamplingParams (#45848)
    • 作者: Will Eaton | +121/-13 | 3 个文件

    Python SamplingParams is a msgspec.Struct(omit_defaults=True), so an encoded EngineCoreRequest carries sampling_params as a sparse map that drops every field left at its default. The Rust mirror EngineCoreSamplingParams only had #[serde(default)] on a subset of fields, so decode_msgpack:: died on the first omitted-but-required field: The failure is in scalar sampling-params …

  • 295232a #44382 — [Rust Frontend] Add /abort_requests endpoint (#44382)
    • 作者: Sahil Singh | +145/-0 | 3 个文件

    Adds a POST /abort_requests to Rust frontend, contributing to “RL / admin / lifecycle APIs” in the Rust frontend roadmap (#44280). Cancels one or more in-flight requests by ID: - Accepts a JSON body returns 200 OK, mirroring the Python frontend’s /abort_requests contract. - Registered in the dev-mode build_router_with_dev_mode, alongside the other Rust admin routes (/sleep, /collective_rpc, /reset…

  • 56e4345 #44938 — [Rust Frontend] Support prompt-only completions (#44938)
    • 作者: Reid | +550/-27 | 3 个文件

    Part of https://github.com/vllm-project/vllm/issues/44280. The Rust OpenAI completions route rejected max_tokens=0 unconditionally. Python vLLM supports the prompt-only completions case when echo=true, where the API returns the prompt without exposing any generated completion tokens. This PR adds that Rust compatibility path for /v1/completions: - Allows echo=true with max_tokens=0. - Rejects max_…

  • 4c62663 #45744 — [M3] Enable FP8 sparse GQA (#45744)
    • 作者: Thien Tran | +179/-178 | 10 个文件

    Reland of #45680 after M3 is merged Add support for FP8 sparse GQA on NVIDIA - Q is not quantized, only KV is - Update the fused QKNorm+RoPE+insert to support FP8 KV cache - Prefill kernel: MSA for sm100, Triton otherwise - Decode kernel: Triton Note: –attention-backend TRITON_ATTN must be used since FLASH_ATTN doesn’t support FP8 KV cache, and FlashInfer FP8 doesn’t support page size 128 Note: F…

  • 5bdc01b #45743 — [M3] Tune Triton indexer score decode for spec-decode (#45743)
    • 作者: Thien Tran | +72/-49 | 3 个文件

    Reland of #45665 after M3 is merged Currently, to support spec-decode case, we simply launch extra CTAs for each decode token. This is simple, because the kernel remains mostly unchanged. However, it is not good because we could have batched multiple Query tokens together for QK MMA to improve efficiency. This PR does exactly just that. - Before: launch 1 CTA per token across all heads. Each C…

  • 20a5f8b #45232 — [FlexAttention] make custom mask mods fully cudagraphable (#45232)
    • 作者: liangel-02 | +98/-1 | 2 个文件

    sumary previously i added https://github.com/vllm-project/vllm/pull/37692 which allowed users to pass in a custom mask mod, but after https://github.com/vllm-project/vllm/pull/36298 landed enabling full cudagraphs, we need to change where we build the masks. instead of building in the forward per layer, we do so before in the build() function that creates FlexAttentionMetadata same thing for t…

  • 14b438a #45868 — [ModelRunnerV2] Various model/config compatibility fixes (#45868)
    • 作者: Nick Hill | +18/-4 | 4 个文件
    1. Pass input_ids as None in input_embeds case (required by some models impls like transformers backend) 2. Some models expect intermediate_tensors forward arg, and don’t have a default 3. Fix MM is_embed accumulation (was overwriting instead of OR-ing) 4. Fix Whisper with full cudagraph (use padded num_reqs count) 5. Fall back to MRV1 for external_launcher + PP case (for now at least) Found from …
  • 556b063 #44468 — [XPU] Fix test_spec_decode_logprobs: use FLASH_ATTN for XPU in GPU_DETERMINISM_KWARGS (#44468)
    • 作者: Chaojun Zhang | +9/-7 | 2 个文件

    Use FLASH_ATTN for XPU in GPU_DETERMINISM_KWARGS to avoid logprob mismatch beyond abs_tol=0.1 in spec-decode tests. - tests/v1/sample/test_logprobs.py: rename ROCM_DETERMINISM_KWARGS → GPU_DETERMINISM_KWARGS, add XPU→FLASH_ATTN case - .buildkite/intel_jobs/misc_intel.yaml: enable test_logprobs.py on XPU CI

  • 5208287 #42656 — Apply LRU policy only to proper cache entries (#42656)
    • 作者: Stan Wozniak | +29/-56 | 3 个文件

    During execution vLLM not only allocates full blocks, which are reused through caching mechanism, but also partial blocks that aren’t assigned any hash, and thus won’t ever be reused by the caching mechanism. Currently both types of blocks are treated equally by the LRU policy. This means that partial blocks, which aren’t in practice “cache” entries, are treated as if they were standard cache entr…

  • d8d9599 #44558 — [Core] Add prefill step cadence for better non-PD DP balancing (#44558)
    • 作者: Nick Hill | +306/-4 | 7 个文件

    Found that interval=2 was optimal, but that a guard is needed to disable the throttling when saturated, to avoid overall perf degradation in that case. — Setup: DeepSeek-R1-0528 FP4 (671B, MLA, FP8 KV), DP=8 / TP=1 / –enable-expert-parallel, 2× GB200 (4 GPU each), internal LB, cudagraph mode. vllm-bench (Rust) random dataset, Poisson (burstiness=1.0), –ignore-eos. interval=1 = base…

  • f2beaa8 #45725 — [ROCm][Quant] mxfp8 moe/linear gfx950 tuning for MiniMax-M3 (#45725)
    • 作者: Hongxia Yang | +39/-7 | 2 个文件

    native mxfp8 moe/linea tuning to improve perf The native MXFP8 dot_scaled MoE/linear kernels used one hardcoded tile config (block_m=64, BLOCK_N=128, num_warps=8), which under-tiles long prefills. Gate the tiles on token count (threshold 1024): - prefill (>=1024): block_m=128, BLOCK_N=256, num_warps=8, num_stages=2 - decode (<1024) : block_m=64, BLOCK_N=64, num_warps=4, num_stages=2 **Serve (ident…

  • 44b2512 #45767 — [KV Connector][Mooncake] Add cache_prefix to namespace store keys (#45767)
    • 作者: Dao007forever | +43/-0 | 4 个文件

    MooncakeStoreConnector keys are derived from the model name, parallel ranks, KV cache group, and block hash. Multiple deployments pointed at the same Mooncake master therefore collide on identical block hashes — one deployment can serve another’s cached KV, and a reset_prefix_cache in one wipes the shared pool for all of them. The motivating case is a **rollout where two deployments of the sam…

  • 188c687 #45488 — [KVConnector][MoRIIO] Allow overriding the advertised host IP (#45488)
    • 作者: kourosh hakhamaneshi | +30/-4 | 3 个文件

    Add host_ip to mori connector’s extra kwargs so that we can pass in routable ips from orchestrator layer for cross node PD.

  • 89e8645 #45637 — [Model] Remove Dots1ForCausalLM (#45637)
    • 作者: Tiezhen WANG | +1/-562 | 4 个文件

    Dots1ForCausalLM (RedNote dots.llm1, MoE text-gen model added in PR #18254, June 2025) has seen modest production usage. Per vllm-release telemetry over the last 6 months: 45 instances / 99.06 GPU-hours. The sibling DotsOCRForCausalLM (rednote-hilab/dots.ocr, separate impl file dots_ocr.py) is unaffected by this removal. Adds Dots1ForCausalLM to _PREVIOUSLY_SUPPORTED_MODELS so users get a clear er…

  • 88a9cdd #45461 — [Model Runner V2] Enable GraniteMOE for MRv2 by default (#45461)
    • 作者: Wentao Ye | +2/-1 | 2 个文件

    Part of https://github.com/vllm-project/vllm/issues/41286 We enable GraniteMOE as a next step because their are plenty of tests for it Covered in CI

  • 506ec6d #45793 — Upgrade tpu-inference to v0.22.1 (#45793)
    • 作者: Sting Lin | +1/-1 | 1 个文件

    Upgrade tpu-inference to latest stable release v0.22.1 Verified on tpu-inference CI. ## Test Result Success. —

  • a52205b #43098 — [Model] Add HrmTextForCausalLM (Hierarchical Reasoning Model — Text) (#43098)
    • 作者: yifei wu | +658/-0 | 8 个文件

    HRM-Text shipped in transformers 5.9.0 (released 2026-05-20, merged to main via huggingface/transformers#46025). The model performs a hierarchical recurrent forward over two transformer stacks (H slow, L fast) inside nested H/L cycle loops; each recurrence step consumes a distinct KV cache slot, matching transformers’ cycle_offset formula. Attention uses a sigmoid gate (Qwen3Next-style) and a fuse…

  • 3d34f8c #44178 — [ROCm][Cleanup] Remove stale AITER FA hybrid KV-cache TODO (#44178)
    • 作者: Tuukka Sarvi | +0/-2 | 1 个文件

    Remove a stale TODO from the ROCm AITER FA shuffled KV-cache update path. The TODO said correct hybrid-model KV-cache handling still needed to be added because Mamba state could make the KV cache non-contiguous. Since I think #43660, the ROCm AITER FA backend uses block-first KV-cache layout and the shuffled writer uses the actual key/value cache block strides, so the TODO no longer describes the …

  • eb04c76 #43050 — feat: MLA prefill enable FA4 fp8 output (#43050)
    • 作者: Carl Y | +489/-20 | 13 个文件

    Completes FlashAttn x Static FP8 in https://github.com/vllm-project/vllm/issues/35792 - Enables fused static FP8 output in FA4 backend - currently points at https://github.com/vllm-project/flash-attention/pull/135 ## Test Result Eval —

  • cca3365 #45753 — [Rust Frontend] Add CORS support (#45753)
    • 作者: Tahsin Tunan | +732/-42 | 12 个文件

    Summary - Adds CORS to the Rust frontend (tracking: #44280), matching the Python frontend’s Starlette CORSMiddleware. - A tower_http::cors::CorsLayer is applied to every response with the same permissive defaults Python uses (allow_origins=[""], methods/headers [""], credentials off), plus a small middleware that keeps non-CORS responses clean. ### Notes - CorsLayer reproduces Starlette’s br…

  • a8c86ee #45306 — [Quant] Support modelopt_mixed on Ampere (SM80/SM86) (#45306)
    • 作者: Mike G | +7/-1 | 1 个文件

    Support modelopt_mixed on Ampere (SM80/SM86) Enables modelopt_mixed checkpoints (NVFP4 routed experts + FP8 weight-only dense layers) on Ampere — A100/SM80 and RTX 30-series/SM86. ModelOptMixedPrecisionConfig rejected these GPUs at engine-config validation (“Minimum capability: 89”), even though modelopt_mixed does not require native FP8 tensor cores. Its layers run on Marlin, which needs only c…

  • 405c7cf #42726 — [ZenCPU] Add zencpu Platform Runtime Logging and Docs (#42726)
    • 作者: Lalithnarayan C | +108/-3 | 6 个文件

    Change-Id: I6049a7950a6c312c02892a740576411e886271a7 Add AMD Zen CPU runtime visibility to the public fork so it is easy to confirm, from live logs, that: - ZenCpuPlatform was activated - CPU unquantized GEMM dispatch went through the zentorch path This PR also updates unit tests - to verify the ZenCPU platform verification logs - and installation/verification docs for the Zen CPU flow. 1. Run the…

  • d53f459 #44528 — [KV Connector][Mooncake] Pipeline-parallel support for PD-disaggregated serving with Mooncake connector (#44528)
    • 作者: Hank Han | +680/-34 | 4 个文件

    This PR adds Mooncake-specific support for pipeline-parallel prefill in PD-disaggregated serving. It is intended for long-context workloads where the prefill side needs PP to fit or run efficiently on H20-class devices. Decode-side PP is intentionally not part of this PR. ### Unit tests Run the Mooncake unit suites from a Kubernetes validation pod using the PR source checkout: ### GLM-5.1-FP8 E2E …

  • b2cfae7 #45631 — Add Triton recompile detection (#45631)
    • 作者: Thien Tran | +81/-15 | 6 个文件

    Many Triton kernels in vLLM recompile during runtime for various reasons: - tl.constexpr args - Pointer alignment: divisibility by 16 or not - Stride alignment: divisibility by 16 or not (or more precisely, making certain 2D tl.load() pattern have 16-byte alignment) The last 2 are not obvious when looking at a kernel. Usually in vLLM we can lose alignment because of tensor slicing (i.e. mixed pref…

🐛 Bug Fix

  • 5e27b2b #45917 — [Bugfix] Pass TP group to FlashInfer all-reduce fusion (#45917)
    • 作者: danisereb | +2/-8 | 2 个文件

    Symptom Serving Nemotron-3-Ultra-NVFP4 with –tensor-parallel-size > 1 and –data-parallel-size > 1 hangs at startup during CUDA-graph capture / warmup: workers freeze and EngineCore repeatedly logs No available shared memory broadcast block found in 60 seconds. The server never becomes ready. Bug reproduced on both H100 and GB200. Using TP>1/DP=1, TP=1/DP>1, and TP=1/DP=1 works (with expert p…

  • eb0fdeb #45831 — [Bugfix][PD] Fix DSV4 disaggregated serving (#45831)
    • 作者: zhanqiuhu | +4/-1 | 1 个文件

    Bug Recent changes broke DSV4 PD disagg. The is_mla_region check in base_worker.py only recognizes MLAAttentionSpec, but DSV4’s compressor layers are SlidingWindowMLASpec. ### Fix Include SlidingWindowMLASpec in the isinstance check so these regions are treated as MLA region. ### Test ### - DSV4 Flash 4P4D on 8xH100 starts and serves requests after this fix. - PR #42310 adds test cases for DSV…

  • d537122 #45782 — [ROCm][Bugfix]: Fallback GFX942 sparse MLA ops to Triton (#45782)
    • 作者: vllmellm | +15/-34 | 1 个文件

    This PR resolves severe generation accuracy degradation, numerical instability occurring in sparse MLA-based models on ROCm platforms. During our internal evaluations on the main branch, we identified a critical regression: - GLM-5.1-FP8 suffered from a massive accuracy collapse (dropping to ~55% on GSM8K). - DeepSeek-V3.2 experienced a slight but noticeable accuracy degradation (~3% drop on GSM8K…

  • f694d43 #45913 — [Bugfix][test] Use Salesforce/wikitext for ppl tests (#45913)
    • 作者: wentian-byte | +1/-1 | 1 个文件

    datasets >= 5.0 + huggingface_hub >= 1.19 no longer accept the legacy single-segment id ‘wikitext’ (HfUriError: ‘Repository id must be namespace/name’). The dataset moved to ‘Salesforce/wikitext’. Without this fix, all PPL tests in tests/models/language/generation_ppl_test/ fail at load_dataset() before any PPL is computed: huggingface_hub.errors.HfUriError: Invalid HF URI ‘hf://datasets/wikitext@…

  • 3c6084b #45852 — [Bugfix][Gemma4] Pre-initialise streaming reasoning state when prompt ends inside an open <|channel> (fixes #45834) (#45852)
    • 作者: Nikhilesh Chhetri | +270/-0 | 6 个文件

    Fixes #45834. After #45553 the Gemma4 tool chat template inserts <|channel>thought\n into the generation prompt when continuing a turn after a tool response with enable_thinking=True. The prompt token IDs therefore end inside an open reasoning channel (a <|channel> start with no matching <channel|> close). Gemma4Parser’s engine, however, always starts in ParserState.CONTENT. As a result the fi…

  • 68ff30d #42332 — [Bugfix] Fixes MiniCPM-O resampler device placement to avoid tensor device mismatch (#42332)
    • 作者: Joel Smith | +3/-1 | 1 个文件

    Fixes #42322. MiniCPM-O 2.6 crashes on startup with Expected all tensors to be on the same device. This is caused by Resampler2_5._set_2d_pos_cache creating the pos_embed buffer on CPU during init (found in minicpmv.py). Typically, it would be moved to GPU via MiniCPMVBaseModel._ensure_resampler_device which is generally called during the load_weights of MiniCPMVBaseModel. MiniCPMOBaseModel overri…

  • e9993a5 #45897 — [BugFix][CI] Fix scheduler plugin test (#45897)
    • 作者: Nick Hill | +1/-1 | 1 个文件

    Fix test breakage from https://github.com/vllm-project/vllm/pull/44558, must have not run in the PR’s CI.

  • a46abb7 #45312 — [Bugfix][Quantization] Reject unsupported compressed tensors KV cache schemes (#45312)
    • 作者: Ting SUN | +1/-1 | 1 个文件

    CompressedTensorsKVCacheMethod.validate_kv_cache_scheme uses and when checking the supported KV cache scheme: That only rejects schemes where both fields are wrong. Half wrong schemes such as {“type”: “float”, “num_bits”: 4} and {“type”: “int”, “num_bits”: 8} pass validation even though the validator message says the supported scheme is num_bits=8, type=float. The fix changes the condition to or, …

  • 7b5d60c #45195 — [Bugfix][V1] Clean up compiled-model bytecode hooks on VllmRunner exit (#45195)
    • 作者: Ting SUN | +61/-2 | 4 个文件

    This is a rebased continuation of #35676, originally opened by @zou3519. The old PR is still open, but the thread confirmed that opening a new PR is the preferred handoff. VllmRunner can run V1 in-process with VLLM_ENABLE_V1_MULTIPROCESSING=0. In that mode, compiled models can stay pinned after runner exit because TorchCompileWithNoGuardsWrapper registers a global TorchDynamo bytecode hook with a …

  • 2785a5e #44912 — [Bugfix][ROCm] Fix FP8 per-tensor scale rank mismatch causing Inductor assertion failure (#44912)
    • 作者: nehmathe2 | +6/-0 | 1 个文件

    Fix AssertionError in Inductor’s aten._scaled_mm lowering due to mismatched scale tensor ranks when AITER is enabled (VLLM_ROCM_USE_AITER=1) and skinny GEMMs are disabled ( VLLM_ROCM_USE_SKINNY_GEMM=0) on ROCm. requantize_with_max_scale returns a 0-D weight scale, but AITER’s _rocm_aiter_per_tensor_quant_fake always returns a 1-D activation scale. When torch.compile lowers aten._scaled_mm, it asse…

  • efd15e1 #45720 — [Bugfix][ROCm] Fix MiniMax-M3 FP8 KV cache dtype (#45720)
    • 作者: Cameron Quilici | +60/-5 | 3 个文件

    Fixes #45562. MiniMax-M3’s sparse-attention backend reinterprets the byte-backed FP8 KV cache as torch.float8_e4m3fn for every E4M3 configuration. That is incorrect on ROCm gfx942, where current_platform.fp8_dtype() is torch.float8_e4m3fnuz. FN and FNUZ use different encodings. Reinterpreting FNUZ cache bytes as FN changes the K/V values before the sparse-attention kernels consume them. The prefil…

  • b831374 #45832 — [Bugfix][Gemma4] Fix parsing when thinking is disabled (#45832)
    • 作者: Federico | +70/-28 | 2 个文件

    Gemma4Parser.adjust_request (added in #45553) returns early when thinking is disabled (enable_thinking=False), so skip_special_tokens is never set to False. With thinking disabled and tools active, the <|tool_call> delimiters get stripped before the parser sees them, so tool calling breaks: tool_calls is empty and the raw call:fn{…} body leaks into content, both streaming and non-streaming. The …

  • 71bc19d #45589 — [Bugfix] Fix MoE model load OOM in FlashInfer_TRTLLM backend with sleep mode (#45589)
    • 作者: Dakai An | +43/-32 | 2 个文件

    This PR fixes the load-time OOM reported in #43951 for large BF16 MoE models using –enable-sleep-mode with the FlashInfer TRTLLM MoE backend. Instead of bypassing the max_split_size_mb=20 allocator policy introduced by #41268, this PR reduces the transient GPU memory created by FlashInfer TRTLLM BF16 MoE weight conversion itself. The conversion now preallocates the final block-layout output t…

  • b9684d9 #45795 — [Bugfix] Gemma4: skip forced JSON for required/named tool choice (#45795)
    • 作者: Federico | +120/-1 | 2 个文件

    Fix a regression from #45588: with the engine-based Gemma4 parser (supports_required_and_named=False), tool_choice=“required” and named tool choice return the forced FunctionDefinition JSON as content with an empty tool_calls list, both streaming and non-streaming. Root cause: the base ToolParser.adjust_request constrains the model to that JSON via structured outputs, but the native Gemma4 parser …

  • 7d56717 #45763 — [Bugfix] Fix Qwen3 prompt tool-call reasoning false positive (#45763)
    • 作者: alexbi29 | +40/-0 | 2 个文件

    Fix a Qwen3 streaming reasoning false positive introduced by the engine-based parser path. When the prompt contains tool-call examples, the prompt token IDs can include <tool_call> before the final generation token. Qwen3Parser.is_reasoning_end() scanned backward for an unpaired <tool_call> and returned True, so DelegatingParser.parse_delta() treated reasoning as already ended before gener…

  • c45f681 #45438 — [Bugfix][Core] Fall back when numactl –membind is blocked in constrained containers (#45438)
    • 作者: Ting SUN | +82/-0 | 2 个文件

    Explicit NUMA binding (–numa-bind with –numa-bind-nodes or –numa-bind-cpus) writes –membind into the numactl wrapper without checking whether the container can set a NUMA memory policy. In a constrained container with set_mempolicy blocked, numactl exits before it execs Python, so the worker or EngineCore subprocess dies with exit code 1 and no Python traceback. This is the same failure report…

  • bf5149b #36616 — [Bugfix] Fix FlashMLA sparse accuracy with topk_length and zero-init padding (#36616)
    • 作者: Ajay Anubolu | +13/-3 | 1 个文件

    Pass topk_length to flash_mla_sparse_fwd for precise attention masking and use new_zeros instead of new_empty for BF16 head padding. Closes #36524

  • c5e5c33 #45707 — [Bugfix][MoE] Restore routed output unpadding before shared expert add (#45707)
    • 作者: Netanel Haber | +13/-18 | 2 个文件

    Fixes regression for https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 and https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 running with FLASHINFER_TRTLLM MoE. Regression introduced in https://github.com/vllm-project/vllm/pull/41184. The test that would have caught this was skipped and wasn’t reenabled when FLASHINFER_TRTLLM was enabled for nano nvfp4: ht…

  • ebf3a6d #45307 — [Bugfix] Fix trtllm fused allreduce+rms_norm for transformers backend (#45307)
    • 作者: Thomas Parnell | +7/-0 | 1 个文件

    The following currently crashes on main: gives: See reproducer above. ## Test Result Works fine after fix. —

  • 7ad894c #44784 — [Bugfix] Prevent cuMemcpyBatchAsync segfault with MTP and KV offloading (#44784)
    • 作者: joshua abraham | +744/-5 | 2 个文件

    Bug: Currently, vLLM crashes with a cuMemcpyBatchAsync segfault when using native KV offloading (–kv-offloading-backend native) alongside MTP speculative decoding on long contexts. Root cause: I found that the OffloadingConnectorScheduler was accidentally including EAGLE/MTP draft-attention groups in both the load and store scheduling paths. Since these draft groups don’t have stable bloc…

🖥️ Kernel

  • 46f74e1 #34432 — [Kernel][Helion][1/N] Add Helion kernel for rms_norm_dynamic_per_token_quant (#34432)
    • 作者: Xiaohong (Sean) Chen | +6750/-0 | 4 个文件

    This PR is to add Helion kernel for operation. It follows the implementation from the vllm c version. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. Autotuning and benchmarking in H200 and B200 were contributed by @gmagogsfm. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: a…

  • 93bbe94 #41430 — [Kernel] Add weightless RMSNorm CUDA kernels for has_weight=False (#41430) (#44109)
    • 作者: arghyadeep sarkar | +231/-117 | 11 个文件
  • 4bf699d #45473 — [Kernel] Support DS Mamba tail copy for MTP align mode (#45473)
    • 作者: sungsoo ha | +182/-34 | 3 个文件

    Add support for speculative decoding with mamba_cache_mode=align when Mamba conv state uses DS layout ([num_blocks, dim, state_len]). For DS layout and accepted-token offset > 0, the conv-state tail is strided and cannot use the existing contiguous memcpy path. This PR adds a DS-tail copy spec and a Triton copy kernel for: The kernel keeps pointer offset arithmetic in int64 to avoid overflow for l…

  • 9d4dc4c #43525 — [Kernel] Support GLM-5 dimensions for TRT-LLM ragged MLA prefill (#43525)
    • 作者: Mohammad Miadh Angkad | +200/-77 | 10 个文件

    This PR lets TRTLLM_RAGGED MLA prefill accept GLM-5 dimensions: - DeepSeek-R1: (qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128) - GLM-5: (qk_nope_head_dim=192, qk_rope_head_dim=64, v_head_dim=256) It also replaces the old DeepSeek-R1-only validation flag with per-backend supported MLADimensions declarations, so FLASHINFER and TOKENSPEED_MLA stay limited to (qk_nope_head_dim=128, qk_rope…

  • ce3ef17 #36895 — [Kernel][Helion][1/N] Add Helion kernel for rms_norm_per_block_quant (#36895)
    • 作者: Xiaohong (Sean) Chen | +8573/-0 | 5 个文件

    This PR is to add Helion kernel for operation. It follows the implementation from the vllm c version. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: and Autotuning Setup Default “full” autotuning effort. i.e. LFBOTreeSearch…

⚡ Performance

  • fa85ead #41992 — [MM][Perf][CG] Support ViT full CUDA graph for Kimi-VL (#41992)
    • 作者: Oğuzhan KIR | +498/-39 | 5 个文件

    Metric | No CG | With CG | Δ – | – | – | – Mean TTFT | 132.91 ms | 111.13 ms | ↓ 16.4% Median TTFT | 130.88 ms | 102.64 ms | ↓ 21.6% P99 TTFT | 290.47 ms | 193.25 ms | ↓ 33.5%

  • 8e27a9c #44944 — [PERF] Fuse multi-group block table staged writes (#44944)
    • 作者: Song Zhixin | +242/-17 | 3 个文件

    Add a fused Triton path for V2 BlockTables.apply_staged_writes so multi-KV-group updates are applied with a single kernel instead of one kernel per block table. Keep the single-group path on the existing StagedWriteTensor implementation. Add CUDA coverage for multi-group writes, hybrid block ID expansion, num_blocks synchronization, staged-write cleanup, and the single-group fallback path. ### Sta…

  • ced32bb #42425 — [Perf] Add VLLM_TRITON_FORCE_FIRST_CONFIG to skip Triton autotuning (#42425)
    • 作者: Francesco Fusco | +209/-0 | 4 个文件

    Triton’s @triton.autotune benchmarks every candidate config on the first launch per key, then caches the winner. Three things make this painful when debugging or measuring: 1. Autotune is timing-driven, so its winner is non-deterministic. Run-to-run jitter can promote a different (BLOCK_M, num_warps, …) tuple each time, and a different tuple reduces partial sums in a different order — so ide…

  • ad32608 #43586 — [MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR (#43586)
    • 作者: Shanshan Shen | +809/-69 | 16 个文件

    This PR implements the SupportsEncoderCudaGraph protocol for DeepseekOCRForCausalLM, enabling full CUDA graph capture of the vision encoder with a dual-path graph architecture. DeepSeek-OCR uses a two-tower ViT (SAM + CLIP) with a dynamic tiling mechanism — global images at 1024×1024 and optional local patches at 640×640. Rather than capturing a single monolithic graph for both paths, this PR …

🔩 Misc

  • 43fa24e #45873 — [Misc] Validate Cohere Embed Mixed Content Payloads (#45873)
    • 作者: Taneem Ibrahim | +53/-0 | 2 个文件

    The purpose of this PR is to make Cohere /v2/embed mixed-content requests fail early and cleanly when a payload is malformed. Currently, payloads like {“type”: “text”} or {“type”: “image_url”, “image_url”: {}} would pass request parsing, then _mixed_input_to_messages() would skip the invalid part because it only appended valid text or valid image_url items. In some cases, the request would silen…

  • 475a6ad #45853 — [Misc] Update Mergify tool-calling label (#45853)
    • 作者: Flora Feng | +4/-0 | 1 个文件

    PRs that touch vllm/parser/, vllm/reasoning/, tests/parser/, or tests/reasoning/ are closely related to tool-calling but weren’t getting the tool-calling label automatically. This adds those directories to the Mergify labeling rule so they’re tagged consistently.

  • 3f1ff1f #45792 — [Misc]Clean up useless test (#45792)
    • 作者: wangxiyuan | +0/-25 | 1 个文件

    seed_everything platform api has been removed already. Related test code can be cleaned up as well. ## Test Result —

🧪 CI/Tests

  • d78650c #45843 — [CI][NIXL] Pin NIXL to 1.2.0 (#45843)
    • 作者: Itay Alroy | +1/-1 | 1 个文件

    NIXL 1.3.0 breaks some vLLM CI tests, so pin NIXL to 1.2.0 until these are resolved. cc: @tlrmchlsmth

  • aa0ac8a #45865 — [CI] Run pre-commit on self-hosted vllm-runners (#45865)
    • 作者: Kevin H. Luu | +10/-1 | 2 个文件

    Run the pre-commit job on the self-hosted vllm-runners runner group instead of GitHub-hosted ubuntu-latest: The lightweight pre-run-check gate stays on ubuntu-latest, so the hook-running job only lands on self-hosted runners after the trusted-author/label gate passes. It also installs shellcheck via shellcheck-py so tools/pre_commit/shellcheck.sh skips its wget + tar -xJ self-download, which t…

  • ef2c40d #45870 — [XPU][CI] fix server test file path (#45870)
    • 作者: Kunshang Ji | +1/-1 | 1 个文件

    intel ci daily/nightly fail https://buildkite.com/vllm/ci/builds/72535#019ed23d-66a9-4926-88db-3387f0466c1a fix test file path ## Test Result —

  • 4fadf9c #45858 — [ROCm][CI] fix multimodel run cmds (#45858)
    • 作者: Divakar Verma | +5/-7 | 1 个文件

    Fixes the mismatch between the test naming and the actual test being run: Multi-Modal Models (Extended Generation 1) was mistakenly running the language generation tests

  • 6f612fb #45722 — [ROCm][CI] Patch conftest to resolve occasional OOMs (#45722)
    • 作者: Micah Williamson | +15/-0 | 1 个文件

    Sometimes we see OOMs in CI when a test starts while a previous test is lazily releasing memory. This PR should work around that case by waiting for memory to settle before starting a test. Here is an example of the failing case: https://buildkite.com/vllm/ci/builds/70756/canvas?jid=019eabf3-e26f-4150-af1e-d7ea834f1278&tab=output#L660

  • 040df8f #45728 — [CI] Fix attention benchmark smoke test (#45728)
    • 作者: Matthew Bonanni | +2/-2 | 2 个文件

    Forward fix for the CI failure that prompted the revert in #45486. — AI assistance (Claude) was used to prepare this change.

  • 7e179e4 #41532 — [ROCm][CI] Gate incompatible HF references on Transformers v5 (#41532)
    • 作者: Andreas Karatzas | +217/-145 | 10 个文件

    This updates generation-test metadata for models whose HF reference path is incompatible with the installed Transformers v5 runtime. Test groups fixed: - mi355_1: Language Models Tests (Standard) for MiniCPM4. - mi355_1: Language Models Test (Extended Generation) for HyperCLOVAX. MiniCPM4: - Fails in HF remote code because it imports is_torch_fx_available, removed from the installed Transformers v…

  • c69c734 #44372 — [XPU][CI] add intel xpu cases for nightly CI (#44372)
    • 作者: wenjun liu | +119/-0 | 2 个文件

    Add xpu cases for nightly in CI pipeline. PR ## Test Result —

🦀 Rust Frontend

  • f00e163 #45701 — [Frontend] Add Streaming Parser Engine and new MinimaxM2 Parser (#45701)
    • 作者: Chauncey | +588/-481 | 12 个文件

    [Frontend] Add Streaming Parser Engine and new MinimaxM2 Parser ## Test Result this pr main —

  • c4fd979 #45759 — [Frontend] Remove AsyncMicrobatchTokenizer. (#45759)
    • 作者: wang.yuqi | +12/-219 | 2 个文件

    Remove AsyncMicrobatchTokenizer. 1. AsyncMicrobatchTokenizer uses a 2ms wait_timeout, which is too large for preprocessing, as the normal time for most tokenizer.encode calls is ~0.1ms. This has a significant impact on Frontend preprocessing, especially for pooling models and the render API. 2. The batch acceleration of tokenizer.encode comes entirely from multi-threading; without TOKENIZERS_PARAL…

🔧 Refactor