共 66 个 commit，涉及 335 个文件，+22593/-2191 行变动。

概要

统计项 数值
Commit 数 66
变更文件 335
新增行数 +22593
删除行数 -2191

Commit 列表

📦 Other

  • 75bdad4 #48507 — [Bug][Quantization] Fix humming is_layer_skipped for compressed-tensors “re:” ignore entries (#48507)
    • 作者: Zhongdongming Dai | +67/-2 | 2 个文件

    HummingConfig.is_layer_skipped decides which layers stay unquantized. It reads the checkpoint’s ignore / ignored_layers / modules_to_not_convert list and matches each entry against the layer prefix using a plain substring check: compressed-tensors checkpoints express ignored layers as regex patterns prefixed with re: (e.g. “re:vision_tower.*”, r"re:.mlp.(gate|up|gate_up|down)_proj."). T…

  • 7cd1d57 #47442 — [CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds (#47442)
    • 作者: Artem Perevedentsev | +8/-120 | 7 个文件

    In nvidia-cutlass-dsl wheel package there was a bug in using the same paths in filesystem by two different sub-wheels that resulted in race conditions during uv pip install process in vLLM. Original bug reports https://github.com/NVIDIA/cutlass/issues/3259, https://github.com/NVIDIA/cutlass/issues/3170 in cutlass repository. To overcome this issue vLLM used some temporary workarounds https://githu…

  • d803b44 #47559 — [NIXL] Bump nixl to 1.3.1 (#47559)
    • 作者: ovidiusm | +1/-1 | 1 个文件

    Bump NIXL version to 1.3.1. requirements/kv_connectors.txt is in run_all_patterns, so editing it triggers the full NixlConnector P/D sweep; the -nixlconnector-pd-accuracy- jobs exercise the NixlConnector against the new nixl wheel. ## Test Result TBD — awaiting the -nixlconnector-pd-accuracy- sweep on this branch. —

  • 530852f #48481 — [KV Connector] Fix PD async scheduling race condition for hybrid attn models (#48481)
    • 作者: Artem Perevedentsev | +158/-7 | 4 个文件

    IMPORTANT: PR was fully copied from https://github.com/vllm-project/vllm/pull/47373 This PR fixes an accuracy regression in PD disaggregation with async scheduling for hybrid attention + SSM/GDN models. When a decode-side request receives KV blocks from a remote prefill worker, the newly allocated attention blocks are overwritten by the incoming NIXL transfer. For hybrid models, vLLM also sc…

  • a9531ed #48150 — [KV Offload] Define clean backend configuration boundary (#48150)
    • 作者: Chang Guo | +1275/-843 | 27 个文件

    Part of #47929. This PR defines plain offloading configuration structs in vllm/v1/kv_offload and moves the existing KVCacheSpec and KVCacheConfig parsing for block geometry, CPU sizing inputs, and file identity into the offloading connector, which translates the raw configs into those structs. The struct shape (model-, cache-, and parallel-level sub-configs with tokens_per_block, blocks_per_chunk,…

  • 9f8cbfd #48209 — Vectorize prep xfer list creation (#48209)
    • 作者: Ilia Yastrebov | +85/-74 | 3 个文件

    In P/D disaggregation with the NIXL connector, the first KV transfer to each new peer is very slow (up to several seconds), spiking first-token latency. The decode-side handshake (add_remote_agent) is dominated by vLLM Python code that builds the per-block NIXL descriptor list — a Python loop creating ~6.5M (addr, len, dev) tuples — which NIXL then ingests element-by-element. This one-time ~1….

  • ea1d65f #47741 — [Rust Frontend] Add Seed-OSS tool parser (#47741)
    • 作者: Chao-Ju Chen | +332/-18 | 7 个文件

    This PR is the tool-parser half of Seed-OSS support in the Rust frontend. The reasoning side is already there, but the Seed-OSS tool-call wrapper is different from Qwen3-Coder, so a Seed-OSS model can’t return tool calls yet. This finishes it. Roadmap item is seed_oss under #44280. The tool-call format is just Qwen3-Coder with a different wrapper. vllm/parser/seed_oss.py says as much: <tool_call> …

  • f44f3d6 #47965 — [Rust Frontend] Wait for mock engine endpoints before ZMQ connect (#47965)
    • 作者: Reid | +34/-24 | 1 个文件

    The mock engine can race ahead of frontend endpoint binding during startup. For TCP endpoints, it previously attempted the ZMQ connect without any readiness wait. For IPC endpoints, it only waited for the socket path to exist, which does not always mean the listener is ready to accept connections. When the connect happens too early, zeromq treats the refused connection as retryable and backs off b…

  • 85e2969 #47973 — BF16x3 router GEMM (#47973)
    • 作者: Thien Tran | +555/-10 | 5 个文件

    This PR is inspired by CuBLAS BF16x9: https://developer.nvidia.com/blog/unlocking-tensor-core-performance-with-floating-point-emulation-in-cublas/ ## Background All FP32 numbers, including subnormals, can be represented exactly by a (weighted) sum of 3 BF16 numbers. This is because FP32 has 24 mantissa bits (including the implicit 1.xxx for normal numbers) and BF16 has 8 mantissa bits. A simple pr…

  • dc9f845 #48738 — [Rust Frontend] Fix mock engine test shutdown race (#48738)
    • 作者: Reid | +1/-1 | 1 个文件

    The mock-engine abort test can complete its functional assertions and then fail during cleanup with errors Connection reset by peer. The cleanup helper previously closed the client transport before cancelling the mock engine. If the mock engine processed the transport closure first, normal test shutdown was reported as an unexpected I/O failure. Cancel the mock engine before shutting down the clie…

  • 6570c98 #48799 — [Model] Add Inkling model support [1/N] (#48799)
    • 作者: Woosuk Kwon | +12137/-62 | 95 个文件

    Adds the core vLLM model and frontend support for thinkingmachines/Inkling, including the public thinkingmachines/Inkling-NVFP4 checkpoint. This is the first independently mergeable slice of #48768. It keeps the model implementation under vllm/models/inkling and includes the tokenizer, renderer, reasoning/tool parsers, model registry entries, tests, and the Python-only FA4 dependency needed to ser…

  • 8bfd683 #48787 — [Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787)
    • 作者: Michael Goin | +42/-1 | 6 个文件

    Currently running vllm serve RedHatAI/GLM-5.2-NVFP4-FP8 –tensor-parallel-size 4 –spec-model GLM-5.2-speculator.dspark –spec-method dspark –spec-tokens 7 –kv-cache-dtype fp8 will fail on Blackwell since the global –kv-cache-dtype fp8 will apply to both the GLM 5.2 target and the DSpark drafter. We don’t currently have an attention backend that simultaneously supports non-causal attention and …

  • 59b964f #48437 — fix(lora): validate LoRA rank is positive in PEFTHelper (#48437)
    • 作者: ErenAta16 | +19/-0 | 2 个文件

    Problem PEFTHelper.post_init computes vllm_lora_scaling_factor as lora_alpha / r (or lora_alpha / sqrt(r) for rsLoRA) without checking that r is positive. r comes straight from an adapter’s adapter_config.json, parsed before validate_legal() – the class’s own intended validation gate, called right after construction by LoRAWorkerManager._load_adapter – ever runs. Two failure modes: - r ==…

  • ba47bb5 #47669 — Bump flashinfer version to 0.6.14 (#47669)
    • 作者: Ameen Patel | +53/-18 | 6 个文件

    Pick up flashinfer-ai/flashinfer#3615, which fixes the multi-CTA radix top-k sampler stream hang (flashinfer-ai/flashinfer#3610): the kernel’s epilogue resets the software barrier’s arrival counter with no sync against peer CTAs still polling it, so a peer can spin forever and permanently wedge the stream (GPU pinned at 100% util / low power, no Xid, engine frozen). 0.6.14 is the first release con…

  • 3935829 #48802 — [Docs] fix error key name (#48802)
    • 作者: rongfu.leng | +2/-2 | 1 个文件

    Test Result —

  • 915dffa #47060 — [Attention] Mirror Triton KV dtype checks in MLA (#47060)
    • 作者: Mike G | +26/-0 | 1 个文件

    Apply the same native BF16/FP8 architecture validation used by the Triton Attention implementation to Triton MLA. This is the Triton MLA counterpart to architecture validation in PR #43330 (https://github.com/vllm-project/vllm/pull/43330), “Allow native KV cache dtype in Triton cache update.” The patch prevents unsupported KV-cache dtypes from reaching the Triton MLA cache-update path: - Native FP…

  • 81e13a0 #42543 — [Compilation] Skip x.size(dim) in _decompose_size_nodes (#42543)
    • 作者: nemanjaudovic | +69/-0 | 2 个文件

    Fix _decompose_size_nodes function (vllm/compilation/backends.py) crashing when a pytorch size() node appears inside a python slice object in an FX node’s args. More clear explanation of why the bug happens is in comment underneath, this is a summary. Basically, all size nodes have to be decomposed during compilation of fx graph because between sublayers that are compiled as a whole (such as atten…

  • f95e3f0 #44349 — [Tests] Gate Step3VL under Transformers v5 (#44349)
    • 作者: BRIJ RAJ KISHORE | +9/-1 | 1 个文件

    What this PR does / why we need it Adds a Transformers v5 version cap (max_transformers_version=“5.3”) for Step3VLForConditionalGeneration in the model test registry. validate_rope() no longer accepts the ignore_keys parameter above Transformers v5.4, causing test failures when running against Transformers v5. This is listed as an open item in the tracker issue #38379. This mirrors the identi…

  • 9d1c695 #47677 — [XPU] Add DSpark speculative decoding support for DeepSeek-V4 (#47677)
    • 作者: Ma Jian | +454/-6 | 3 个文件

    Add XPU (Intel GPU) support for DSpark semi-autoregressive speculative decoding on DeepSeek-V4. - vllm/models/deepseek_v4/xpu/dspark.py (new): XPU DSpark draft model using HCHeadOp/MHCPostOp custom ops (from vllm-xpu-kernels) instead of tilelang kernels - vllm/models/deepseek_v4/xpu/model.py: Add EagleModelMixin to DeepseekV4Model and SupportsEagle3 to DeepseekV4ForCausalLM; collect aux_hi…

  • 015b032 #48643 — Add giuseppegrossi to rocm label auto cc action (#48643)
    • 作者: Giuseppe Grossi | +1/-1 | 1 个文件

    ##Purpose Adding myself to github action that auto cc’s users for rocm/amd labeled issues. cc @hongxiayang

  • de100ff #48497 — [Docs] Document pooling config resolution (#48497)
    • 作者: Taneem Ibrahim | +59/-4 | 1 个文件

    Pooling configuration precedence is currently described only at a high level and is scattered across model-specific guidance. This docs only change documents the current resolution chain in one place: user-provided –pooler-config fields, supported Sentence Transformers metadata, and architecture or task defaults. It also distinguishes embedding normalization from classification activation, qualif…

  • 61141ed #41934 — [Hardware][XPU] Register batch-invariant kernels for XPU (#41934)
    • 作者: Tomasz Zielinski | +130/-58 | 6 个文件

    This pull request adds partial support for batch invariance on Intel XPU devices. This is a requirement for reinforcement learning on XPU. Unit tests are modified to cover the registered kernels on XPU. ## Test Result All newly added unit tests pass. ### Note The code was co-developed with GitHub Copilot.

  • 05eed72 #48526 — [ROCm] Re-enable cudagraph memory profiling, captured on the current stream (#48526)
    • 作者: peizhang56 | +37/-8 | 3 个文件

    Re-enables cudagraph memory profiling on ROCm (is_cuda() -> is_cuda_alike()) that #48440 disabled as a stopgap, and fixes the decode-throughput regression that motivated that revert. profile_cudagraph_memory() captures throwaway graphs via graph_capture(), which by default allocates a fresh side stream (torch.cuda.Stream(…)). torch’s caching allocator pools free blocks per stream, so the side-st…

  • 5810e88 #47991 — [Model] Add RobertaForTokenClassification / XLMRobertaForTokenClassification (#47991)
    • 作者: Gopala-Krishna Char | +118/-1 | 5 个文件

    vLLM’s model registry supports BertForTokenClassification and ModernBertForTokenClassification for the /pooling token-classify runner, but RoBERTa-family checkpoints have no token-classification entry. RobertaForSequenceClassification/XLMRobertaForSequenceClassification already exist, but a RoBERTa or XLM-RoBERTa checkpoint fine-tuned for token classification (e.g. NER) currently has no matching a…

  • 615834e #47636 — [KVOffload][P2P] Well-known default host/port env vars and per-DP-rank control port (#47636)
    • 作者: liranschour | +413/-61 | 13 个文件

    Give the P2P KV-offload secondary tier (type: “p2p” under TieringOffloadingSpec) the same well-known-port ergonomics the NIXL connector already has, plus per-DP-rank port binding so data-parallel replicas don’t collide. Two commits: 1. Well-known default host/port env vars. Add VLLM_P2P_SIDE_CHANNEL_HOST (default localhost) and VLLM_P2P_SIDE_CHANNEL_PORT (default 5710), mirroring VLLM_NIXL_SID…

  • 1b30ae4 #47873 — [Rust Frontend] Fix flaky tls_handshake_timeout_drops_silent_client test (#47873)
    • 作者: Tahsin Tunan | +12/-6 | 2 个文件

    Summary - Problem: the test manually advanced the paused clock and raced the handshake close against the socket read, making it flaky. - Reproduce: for i in $(seq 20); do cargo nextest run -p vllm-server tls_handshake_timeout_drops_silent_client; done - Fix: drop the manual advance/yield_now, let tokio’s paused clock auto-advance a bare read().await, and assert the close lands at the handshake…

  • 4e04bcb #48034 — [Rust Frontend] Tolerate whitespace before the outer brace in JSON tool-call parsers (#48034)
    • 作者: Tahsin Tunan | +105/-2 | 3 个文件

    Summary - A Rust JSON parser for tool calls was failing when pretty-printed (with newlines/spaces between braces). Python’s json.loads tolerates that whitespace, but the Rust parser didn’t skip whitespace before the closing }. - Fix: add explicit whitespace skipping before both close braces to match Python’s behavior.

  • 9dd2e72 #48206 — fix flaky multi example connector consistency (#48206)
    • 作者: Aarushi Jain | +22/-4 | 1 个文件

    Fixes a flaky failure in tests/v1/kv_connector/unit/test_multi_connector.py::test_multi_example_connector_consistency. The test asserts the exact per-request scheduler event sequence for each connector, expecting on_new_request at index 0 of every window: on_new_request get_num_new_matched_tokens 0 update_state_after_alloc num_blocks=[…] … build_connector_meta However, build_connector_meta is …

  • d119beb #48159 — [ROCm] Add tuned selective_state_update config for AMD MI350 (#48159)
    • 作者: Giuseppe Grossi | +102/-0 | 2 个文件

    Adds tuned selective_state_update (Mamba SSU decode kernel) launch configs for the AMD Instinct MI350X for both the cache_dtype=float16 and cache_dtype=float32 variants, mirroring how the MI355 (#47767), and MI300X (#47945, #47947) configurations were done. vLLM bundles per-device SSU configs for NVIDIA parts (B200, GB200, H100, H200, RTX PRO 6000) and, recently, MI355 and MI300X. On MI350X ge…

  • 12a8057 #48600 — [CI/Build] Split release artifact annotations by type (#48600)
    • 作者: Kevin H. Luu | +25/-24 | 2 个文件

    Split the shared release-artifacts Buildkite annotation into dedicated release-wheels, release-images, and release-manifests annotations. This preserves inline reporting after each artifact is produced while making release outputs easier to scan. The annotation helper now requires an explicit context, preventing future callers from silently adding unrelated artifacts back to a shared section. Dupl…

  • e281ac6 #48554 — [Rust Frontend] Integrate MM audio support (#48554)
    • 作者: Bugen Zhao | +1274/-257 | 18 个文件

    Signed-off-by: Bugen Zhao i@bugenzhao.com Audio support in llm-multimodal was added in https://github.com/lightseekorg/smg/pull/1905. Similar to #47959, this PR adopts that to further extend the Rust frontend to support audio input and preprocessing. Related code is generalized more a bit to support modalities not limited to vision (image / video). End-to-end test with Qwen3-ASR ## Test Result T…

⚡ Performance

  • d08eeba #47156 — [Perf][MoE] Write FlashInfer combine into final output (#47156)
    • 作者: Samuel Nordmann | +61/-2 | 3 个文件

    Pass vLLM’s preallocated MoE output tensor to FlashInfer one-sided combine, removing the temporary combine output and subsequent output.copy_(). Requires FlashInfer PR: https://github.com/flashinfer-ai/flashinfer/pull/3776 ## Performance Nemotron Ultra 550B A55B NVFP4, one GB200 node (4 gpus), TP1/DP4/EP4, FlashInfer one-sided, ISL/OSL 50000/2048, concurrency 16: | | Baseline | Direct output | Del…

  • b8168e3 #46275 — [ROCm][Perf][DSV4] Enable split sparse decode on gfx942 (#46275)
    • 作者: Tuukka Sarvi | +45/-42 | 2 个文件

    Enable the existing split partial/reduce Triton sparse decode path for DeepSeek-V4 sparse MLA decode on AMD gfx942. gfx950 already uses this path. gfx942 currently falls back to the monolithic _sparse_attn_decode_ragged_kernel, even though the split path runs correctly on gfx942. This PR extends the tuned-architecture guard to include gfx942 and updates the ROCm DeepSeek-V4 sparse attention tests …

  • 3c1bc1f #48519 — [ROCm][Perf] Optimize sparse attention prefill kernel for DeepSeek-V4 (#48519)
    • 作者: kliuae | +10/-3 | 1 个文件

    This PR optimizes DeepSeek-V4’s dedicated sparse attention ragged prefill kernel in ROCm. The optimization speeds up the kernel by ~2x, and in E2E benchmark reduces TTFT by ~10%. In the current upstream ROCm path, this kernel has some slight inefficiencies that can be optimized. Specifically, this PR addresses three points of optimization. - num_warps setting. In DSv4-Pro’s architecture, the per-w…

  • 2dab187 #47463 — [Perf] Optimize fused_topk_bias for DSv4, 1.5~2x kernel performance improvement (#47463)
    • 作者: Wentao Ye | +76/-67 | 3 个文件

    Part of https://github.com/vllm-project/vllm/issues/45861 Originally when hash_indices_table is None we will do a redundant input_tokens.to(dtype=indices_type) which can be removed. We make the conversion inside the kernel to improve performance Acc covered in current unit test Performance using this AI generated script: And we get

  • eb33ff3 #47718 — [ROCm][Perf] DSv4 two-stage compressor kernel for HCA prefill (#47718)
    • 作者: kliuae | +528/-0 | 3 个文件

    This PR adds a two-stage Triton implementation to the DeepSeek-V4 compressor for the prefill path in head_dim=512, cr>=128 to improve TTFT. On upstream ROCm, the single pass compressor-norm-rope-cache fusion is dispatched one workgroup per token. In HCA where compress ratio is 128, only workgroups that live on the CR boundary (positions % 128 == 0) execute on 128x512 grids, and the others early ex…

🦀 Rust Frontend

  • 3e90d01 #47699 — [Frontend] Overlap preprocessing and computation for pooling models offline inference (#47699)
    • 作者: wang.yuqi | +673/-150 | 10 个文件

    TL;DR - Use multithreading to accelerate preprocessing. You can specify the number of threads using renderer_num_workers. - Tiling to overlap preprocessing and computation for pooling models offline inference. - Sadly, because of the GIL, multithreading can only speed up the tokenizer and can hardly accelerate multimodal preprocessing at all. We are looking for ways to use multiprocessing to sp…

🔩 Misc

  • a317bc5 #48717 — [Misc][Nixl] Unify _logical_to_remote_kernel_block_ids (#48717)
    • 作者: Nicolò Lucchesi | +41/-64 | 5 个文件

    Fold _logical_to_remote_kernel_block_ids into _logical_to_kernel_block_ids function + ratio as input, so we can use the same function for both local/remote expansion. cc @ZhanqiuHu

🐛 Bug Fix

  • 8c3393f #48134 — [Bugfix][Rust Frontend] Limit chat top_logprobs in responses (#48134)
    • 作者: Reid | +86/-5 | 4 个文件

    Fixes a Rust frontend chat response parity issue where top_logprobs entries were not truncated according to the requested count. For chat completions with logprobs=true, the Rust frontend previously emitted every decoded candidate returned by the engine for each output token. Python serving limits top_logprobs as follows: - omitted or 0: return an empty top_logprobs list - positive k: return the f…

  • cc706b0 #47707 — [Bugfix][Rust Frontend] Detokenizer: avoid leaking prompt on zero-generated-token completions (#47707)
    • 作者: JinYan Su | +34/-5 | 1 个文件

    DecodeStream::flush() called before any push_token() (i.e. a request that finishes with zero generated tokens — the engine suppressed the only emitted EOS) leaks the entire prompt text into cumulative_output, because seed_prefix never ran and prefix stays empty. This mirrors the Python V1 path, where BaseIncrementalDetokenizer.update() early-returns on empty new_token_ids, leaving output_text unto…

  • 12f2c51 #48530 — [Bugfix] Fix offloading set_ overflow for packed non-uniform KV caches (#48530)
    • 作者: Elvir Crnčević | +8/-7 | 1 个文件
    • When attention layers are packed into a shared KV cache tensor (e.g. MLA models with non-uniform page sizes), each layer’s page_size can be smaller than the block_stride. torch.Tensor.set_(storage, offset, size, stride) validates that offset + size_bytes <= storage_size, but for packed layouts the last block’s offset + page extends past the per-layer boundary into the next layer’s region — valid…
  • df8a090 #48741 — [BugFix] Don’t apply weight in batch-invariant RMSNorm when has_weight=False (#48741)
    • 作者: Asaf Gardin | +21/-13 | 2 个文件

    Fix silent output corruption for models using RMSNorm(has_weight=False) (Gemma4, Gemma3n, Llama4) when VLLM_BATCH_INVARIANT=1 is combined with sleep mode - e.g. in standard RL rollout configuration. RMSNorm(has_weight=False) keeps its identity weight as a plain tensor, not an nn.Parameter - so sleep(level=2) discards it and weight re-sync brings it back zeroed. VLLM_BATCH_INVARIANT branch still pa…

  • 2db39c7 #48068 — [Bugfix][Spec Decode] Fix eagle3 first-layer qkv_proj prefix for quantized drafts (#48068)
    • 作者: qizixi | +1/-1 | 1 个文件

    The EAGLE3 first decoder layer overrides self.self_attn.qkv_proj but builds it with prefix=maybe_prefix(prefix, “qkv_proj”), yielding …layers.0.qkv_proj instead of the canonical …layers.0.self_attn.qkv_proj. For a quantized draft, the quant config matches its exclude_modules against this prefix — the missing .self_attn segment lets the excluded attention get quantized, and the 2 * hidden_size firs…

  • 3a5e88e #48754 — [Bugfix] Fix local speculators with dots in the name from classifying as custom_class (#48754)
    • 作者: Michael Goin | +14/-7 | 1 个文件

    Fix speculative config so draft model names with dots (e.g. local GLM-5.2-speculator.dspark folder) are not misclassified as custom_class import paths. ## Test Result Before: After: loads normally, same as using RedHatAI/GLM-5.2-speculator.dspark for the model —

  • 0becb74 #47309 — [BugFix][MLA] Support kv_cache_dtype_skip_layers for MLA attention (#47309)
    • 作者: ruikangliu | +14/-1 | 1 个文件

    Fix two gaps in MLAAttention that prevent –kv-cache-dtype-skip-layers from working with MLA models (DeepSeek-V2/V3/V4, GLM and others). ### Problem 1: MLAAttention.init ignores kv_cache_dtype_skip_layers The –kv-cache-dtype-skip-layers flag (introduced in #33695) allows skipping FP8 KV cache quantization for specified layers. However, it was only wired up in Attention.init — MLAAttention…

  • 2bd8957 #46880 — [Bugfix][NVFP4 MoE] Pad gated intermediate to 64 for FlashInfer TRT-LLM shuffle (M%128) (#46880)
    • 作者: Mike G | +7/-1 | 1 个文件

    Closes #46879. FlashInfer’s TRT-LLM NVFP4 MoE weight prep shuffles block-scale rows via get_shuffle_matrix_sf_a_row_indices, which asserts the gate/up row dim is a multiple of 128 (epilogue_tile_m). align_fp4_moe_weights_for_fi pads the intermediate to min_alignment, and the gate/up dim is up_mult*padded_intermediate (up_mult=2 when gated). The caller passes min_alignment = 16 if is_gated else 128…

  • ecf4aa5 #48167 — [Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell (#48167)
    • 作者: Michael Goin | +149/-47 | 12 个文件

    DFlash/DSpark drafts on Blackwell had near-zero acceptance or crashed with an illegal memory access when the drafter used FlashInfer; Hopper was fine. Root cause: non-causal attention skips trtllm-gen and runs FlashInfer’s prefill wrapper, which is only CUDA-graph-replayable when constructed with use_cuda_graph=True and persistent buffers — vLLM does not use that mode for draft attention, so repla…

  • b7950e7 #47460 — [Bugfix] Initialize draft CUDA-graph keys for the native draft_model proposer (#47460)
    • 作者: avalliappan-nvidia | +2/-0 | 1 个文件

    Problem When speculative decoding uses the native draft_model method (method=“draft_model”), the draft model runs eager every step — no CUDA-graph replay — even though its PIECEWISE graphs are captured during dummy_run. The engine captures the draft graph but never dispatches to it, so each draft forward is launch-bound (hundreds of thousands of cudaLaunchKernel calls) and can cost **m…

  • 43cd340 #48252 — [Fix] Align OpenAI vllm_xargs value types across request schemas (#48252)
    • 作者: Sage | +4/-4 | 2 个文件

    Some OpenAI request schemas accepted list-valued vllm_xargs while completion requests only accepted scalar values. This aligns the shared vllm_xargs type across completions, chat completions, and responses. split out from #47922

  • 1d99f0f #47770 — [ROCm][BugFix] Triton W4A16 handling for GPTQ/AutoGPTQ qzeros layout (#47770)
    • 作者: Giuseppe Grossi | +182/-5 | 2 个文件

    Fix ROCm Triton W4A16 handling for GPTQ/AutoGPTQ qzeros layout specifically issue #47159. This addresses two related issues: - Symmetric/biased GPTQ models may still have qzeros tensors registered by the loader, but the Triton kernel should not use them. For biased types such as uint4b8, apply_weights() now passes qzeros=None, so the kernel uses scalar zp_bias and runs with HAS_ZP=False. - TritonW…

  • 66b6c68 #48125 — [PD][Bugfix] Fix validation of cache shape for attn backends enforcing different kernel_block_size (#48125)
    • 作者: Nicolò Lucchesi | +6/-1 | 1 个文件

    I have started looking into heterogeneous attn backend selection here https://github.com/vllm-project/vllm/pull/48012 (an extension of hybrid ssm setup if you want). This exposed a latent bug in the validation of cache shapes that we have in the registration phase. Say we have 2 attn backends enforcing different block sizes: - user block_size=128 - FA kernel_block_size=128 - SWA kernel_block_s…

  • c0302d9 #48098 — [Bugfix] Fix parallel_tool_calls=null crash in Responses API from_request() (#48098)
    • 作者: Mahad Rehman | +46/-2 | 2 个文件

    Fixes #48097. Sending “parallel_tool_calls”: null to the Responses API (/v1/responses) caused from_request() to pass None to ResponsesResponse.parallel_tool_calls (typed as bool, non-optional), raising a Pydantic ValidationError during response construction. vLLM’s error handling catches this and returns a 400 with a raw Pydantic error message, instead of resolving to the documented default li…

  • 313fae3 #48711 — [Bugfix] Fix GLM5 config (#48711)
    • 作者: Jee Jee Li | +29/-0 | 1 个文件

    Fix the following error when running nvidia/GLM-5.2-NVFP4on the latest main branch. ## Test Result —

  • 7aab6e2 #48688 — [ROCm][Bugfix] Enable the fp32 head_dtype torch.mm fast path on ROCm (#48688)
    • 作者: Turner Jabbour | +36/-4 | 2 个文件

    #48390 added a torch.mm(…, out_dtype=torch.float32) fast path so the fp32 head_dtype lm_head projection avoids materializing a full fp32 copy of the lm_head weight on every step. That fast path was gated on current_platform.is_cuda(), so ROCm always fell back to the cast path (F.linear(hidden_states.to(fp32), lm_head.weight.to(fp32), …)), which allocates and writes a full fp32 copy of the (lar…

🧪 CI/Tests

  • 7dc2698 #48784 — [ROCm][CI] Set “highest” matmul precision for reference hf_runner in test_bert_for_masked_lm (#48784)
    • 作者: Micah Williamson | +6/-0 | 1 个文件

    This is because the vLLM runner uses matmul precisoin “highest” by default: However, for this test group, we explicitly set torch.set_float32_matmul_precision(“high”) on ROCm to workaround a separate issue (see https://github.com/vllm-project/vllm/pull/31820). This gets picked up by the reference hf_runner, so the accuracy of the reference is less stable than on CUDA which does not set torch.set_f…

  • 6a9f24a #48764 — [ROCm][CI] Fix cuda graph mem profile issue (#48764)
    • 作者: Charlie Fu | +13/-10 | 1 个文件

    Root cause During determine_available_memory → profile_cudagraph_memory, capture runs inside torch.cuda.graph(). PyTorch requires a non-default stream (cuda_stream != 0). On ROCm, the profiling path was using: GraphCaptureContext(torch.cuda.current_stream(self.device)) Before vLLM initializes its dedicated compute stream, torch.cuda.current_stream() is the per-thread default stream (cuda_strea…

  • 7746961 #47375 — [CI] Fix flaky lora test (#47375)
    • 作者: qli88 | +18/-5 | 1 个文件

    Fix tests/lora/test_gptoss_tp.py issue. On ROCm platform the generated string may have less spaces than the expected strings, so add reformatting. Notice: this test group is skipped on ROCm platform now due to Quark version issue. Will be re-enabled when Quark v0.12 is released.

  • 3034c8d #42310 — [CI][PD] Add optional/nightly DSv4 Disaggregated eval (#42310)
    • 作者: Nicolò Lucchesi | +31/-5 | 3 个文件

    As per-title, add DSv4 8 gpus tests. Optional as we don’t want to run this gpu-hungry setup on each PR.

  • 49e777c #48773 — [CI][ROCm] Retry failed Docker build steps once (#48773)
    • 作者: Micah Williamson | +6/-0 | 1 个文件

    Add retry for AMD Docker build in CI

  • 0885b51 #48746 — [CI][ROCm] Stabilize ci_base hash calculation and image handoff (#48746)
    • 作者: Andreas Karatzas | +69/-6 | 2 个文件
    • Calculate the ROCm ci_base content hash three times with five-second intervals. - Require every calculation to succeed and produce the same hash. - Fail when the BASE_IMAGE digest cannot be resolved instead of hashing unknown. - Pass the exact ci_base image selected by the ensure step to the test-image build. Separate Buildkite jobs could calculate different ci_base tags for the same commit. The…
  • 5811ed6 #48545 — [Test][kv_offload] Fix flaky drain() helper in test_fs_tier.py (#48545)
    • 作者: Chaojun Zhang | +4/-18 | 1 个文件

    The drain() test helper in test_fs_tier.py used a polling loop that exited early after 20 consecutive empty rounds (200 ms). Under a loaded system, store tasks could complete after the 200 ms window, causing test_load_job_emits_no_event and test_partially_failed_store_emits_no_event to fail with assert 0 == 1. ## Fix Replace the polling loop with tier.drain_jobs() (which calls wait_idle() and bloc…

✨ New Feature

  • 5de1add #48451 — [feature]Add int4 quantization support for emulation moe backend (#48451)
    • 作者: qli88 | +1653/-10 | 9 个文件

    This PR is to enable int4 quantization support for emulation moe backend. ROCm platform doesn’t have a moe backend that supports asymmetric int4 quantization, so this is necessary to support int4 quantized models like cyankiwi/MiniMax-M3-AWQ-INT4 for now (In the future this support will be added to flydsl/AITER library). Running cyankiwi/MiniMax-M3-AWQ-INT4 on ROCm platform MI300/350/355 to verify…

  • 4238b01 #47881 — [Feature] Migrate moe sp support to non-torch compiled path for GLM5.2 (#47881)
    • 作者: Wentao Ye | +92/-16 | 4 个文件

    We have enabled this for normal path, but there is a need for non torch compiled path, this PR does the migration vllm serve zai-org/GLM-5.2-FP8 –kv-cache-dtype fp8_e4m3 –enable-expert-parallel –tensor-parallel-size 8 –tool-call-parser glm47 –enable-auto-tool-choice –reasoning-parser glm45 –port 9256 –profiler-config.profiler=torch –profiler-config.torch_profiler_dir=/home/yewentao256/pro…

🔧 Refactor

  • 5a65ba5 #46647 — [Refactor] Move iteration logging to the frontend (#46647)
    • 作者: Max Hu | +398/-40 | 9 个文件

    Move iteration-detail logging out of direct EngineCore log emission and into the frontend stats logging path. EngineCore now builds a SchedulerIterationDetails payload and attaches it to the existing SchedulerStats carried by EngineCoreOutputs. The frontend LoggingStatLogger.record() emits the final log line, including existing SchedulerStats.kv_cache_usage. This also records scheduled multimodal …

🖥️ Kernel

  • 6036bf1 #48512 — [Kernel][Helion] Add Helion kernel benchmark script (#48512)
    • 作者: Xiaohong (Sean) Chen | +574/-28 | 6 个文件

    This PR is to add an utility script to benchmark Helion kernels. Major changes are: - Updated all Helion kernels’ baseline function inside kernel impl file to use native torch implementation. This baseline is intended to be used as the baseline for Helion autotuner and correctness unit tests. It is more stable to use native torch impl for this purpose. - Added a script to run ops-level benchmark f…

  • 2fa63e0 #48264 — [Kernel][Helion] Helion kernel lazy registration (#48264)
    • 作者: Xiaohong (Sean) Chen | +31/-6 | 3 个文件

    Current implementation will eagerly register all existing Helion kernels under “vllm/kernels/helion/ops” directory during vLLM server startup even if they are not used. The vLLM server will also log the following message during startup which often confuses user that Helion kernel are used. An ideal approach is to lazily register Helion kernel when it is really used. This PR disabled the eager regi…