共 82 个 commit，涉及 385 个文件，+20644/-3430 行变动。

概要

统计项 数值
Commit 数 82
变更文件 385
新增行数 +20644
删除行数 -3430

Commit 列表

📦 Other

  • 4fb483c #45432 — [Docs] Expand llm-d integration page (#45432)
    • 作者: IBRAHIM IBRAHIM | +34/-2 | 1 个文件

    Expands the llm-d integration page from a two-sentence blurb into a short overview of what llm-d adds on top of a vLLM deployment. The current page only links out to llm-d without explaining what it does. This update gives readers enough context to know whether llm-d fits their use case before clicking through, while keeping it concise and link-forward. ### What’s added - **What llm-d adds to vLLM…

  • 0d0504b #49903 — [Core] Warm up runner-owned Triton kernels before the first request (#49903)
    • 作者: Nick Hill | +183/-189 | 7 个文件

    Several runner-owned Triton kernels only compiled once a real request arrived, spiking first-token latency. Each takes a runtime (non-constexpr) integer that Triton specializes into == 1, % 16 == 0 and neither, so warming a single shape was not enough. - _zero_kv_blocks_kernel is driven by the scheduler’s new_block_ids_to_zero, which no dummy or warmup step reaches. Add KVBlockZeroer.warmup() cove…

  • 601fa9a #49612 — [KV Connector] Support NIXL heterogeneous P/D block sizes for hybrid models (#49612)
    • 作者: Nick Hill | +1014/-111 | 7 个文件

    Hybrid (mamba) models previously asserted out heterogeneous block sizes entirely — yet they are the models where P/D block sizes most readily diverge, since the mamba-padded attention block size varies with TP sharding (#41037). Lift the restriction: - Mamba state blocks are indivisible, so their descriptors always use local page geometry and their desc ids are never ratio-expanded; attention desc…

  • 9b9fc40 #45841 — add epilogue hook to flex attention (#45841)
    • 作者: liangel-02 | +9/-0 | 1 个文件

    adding optional post-attention epilogue transform to flex attention (this is an example of how someone could use this)

  • 98e91a9 #49345 — [PD][NixlPush] Skip extra add_remote_agent step in D->P handshake (#49345)
    • 作者: Nicolò Lucchesi | +6/-3 | 2 个文件

    Address S4 in https://github.com/vllm-project/vllm/issues/48633. In the workflow described here https://docs.vllm.ai/en/stable/design/nixl_kv_push_connector , when D handshakes with P with the purpose of forwarding block_ids for P to write to, D executes a “full handshake”, which includes exchanging kv topology and addresses to register local/remote descs here This is not actually needed to exchan…

  • 35efdf6 #47288 — [Elastic EP] Async preparation (#47288)
    • 作者: Itay Alroy | +701/-677 | 21 个文件

    Async Elastic EP preparation This PR moves Elastic EP preparation off the main serving thread. Creating standby process groups, setting up EP communication, staging quant methods, transferring model weights, synchronizing KV-cache memory, and preparing new workers now happen on background worker threads. Requests continue to be served regularly during this work and are blocked briefly only for t…

  • d2bfc6f #50103 — [Build] Fix DeepEP CUDA driver stub linking (#50103)
    • 作者: Kevin H. Luu | +15/-0 | 1 个文件
    • discover the CUDA toolkit’s libcuda.so driver stub before building DeepEP - prepend its architecture-specific directory to LIBRARY_PATH - fail early with a clear message if the toolkit stub is missing ## Root cause #45321 updated the pinned DeepEP revision to d4f41e4e93. That DeepEP revision adds -lcuda when linking its extension, but the manylinux wheel builders do not contain a real NVIDIA dri…
  • 912d6b6 #47494 — [Rust Frontend] Align sampling validation with Python (#47494)
    • 作者: Reid | +260/-2 | 6 个文件

    Align OpenAI sampling parameter validation with Python. The Rust frontend did not validate several /v1/completions sampling parameters before forwarding requests to the engine. Invalid values such as temperature=5.0, top_p=0.0, min_p=2.0, or frequency_penalty=100.0 could reach Python engine decoding and fail there instead of returning a frontend 400 Bad Request. This PR adds Python-compatible vali…

  • bf9f230 #49496 — [Rust Frontend] Fix finish reason for named tool choices (#49496)
    • 作者: Reid | +88/-17 | 3 个文件

    The Rust chat completions path returned finish_reason: “tool_calls” whenever a response contained a tool call. For a forced named function choice, the expected finish reason is “stop”. The previous behavior caused the Rust and Python frontends to return different terminal reasons for the same request in both streaming and non-streaming modes. Preserve whether the request uses a named tool choice w…

  • 5ed3faa #49992 — [Rust Frontend] Add ordinary-text tokenizer encoding (#49992)
    • 作者: Bugen Zhao | +448/-2 | 10 个文件

    Segment-aware prompt renderers need separate encoding paths for trusted structural markers and literal text. encode_ordinary keeps literal text on the base tokenizer pipeline even when its spelling matches an added token. This PR adds Tokenizer::encode_ordinary, equivalent to encode(text, false) with every added, special, and control-token matcher bypassed. For tiktoken, this directly delegates to…

  • 61ac368 #50090 — [Kimi-K3] Add AttnRes kernels (#50090)
    • 作者: Thien Tran | +1719/-0 | 14 个文件

    #50000. Add AttnRes kernels - Specialized CUDA C++ kernel for NVIDIA sm100 - Triton for NVIDIA fallback and AMD This PR also adds Buildkite CI for K3 ## Test Result —

  • 7458793 #49904 — [Build] Fix CUDA arch detection producing kernel-less builds on SM121 (#49904)
    • 作者: Ayushman Singh | +40/-9 | 3 个文件

    On GB10 / DGX Spark (compute capability 12.1, CUDA 13), this shows up on the default source-build path from the docs, uv pip install -e . with TORCH_CUDA_ARCH_LIST unset. No special flags or custom arch list needed. The build looks fine but has no CUTLASS kernels in it. The .so only contains sm_75 cubins. Serving any FP8 model then fails at startup: The cause is an old line in torch’s vendored sel…

  • a8f2960 #49858 — [KV Offload] Make compact secondary identity TP-independent (#49858)
    • 作者: Chang Guo | +226/-11 | 4 个文件

    This completes the secondary-tier step scoped in #47929. #48906 made the single-node TP-only MLA primary row compact, but secondary-tier identity still included the TP configuration: FS/OBJ paths remained TP-specific, and P2P derived its compatibility fingerprint from the same fields. This PR: - admits replicated_layout rows to the existing caller-opted-in parallel-agnostic FileMapper path; - reta…

  • 33fe71a #46491 — [AMD] Revert Mxfp4MoeBackend.TRITON_UNFUSED fallback (#46491)
    • 作者: fxmarty | +63/-35 | 3 个文件

    (submitting from this account instead of https://github.com/fxmarty-amd as I am hitting You’ve reached the open pull request limit for this repository errors there) This PR reverts a change from https://github.com/vllm-project/vllm/pull/45896, that incorrectly added Mxfp4MoeBackend.TRITON_UNFUSED fallback in MXFP4 MOE. The correct behavior in case VLLM_ROCM_USE_AITER=1 is not specified (or when ru…

  • d18ed23 #49152 — [KV-offload][FS] : Batch store/load_block in C (#49152)
    • 作者: Varun Sundar Rabindranath | +452/-58 | 4 个文件

    We have pools of python threads to read and write KV files from/to disk. These python threads compete for the GIL and submit bursty read/write commands to the disk. This prevents the disk from reaching 100% utilization. ### Changes: 1. Add C implementations for store_block and load_block that does the heavy-lifting inside a GIL-free region. Note that we still use the python thread pool, the optimi…

  • 7aea73d #49348 — [ROCm][Quark][6/N] Use MXFP4 linear kernel abstraction for aiter backend (#49348)
    • 作者: fxmarty-amd | +370/-166 | 6 个文件

    Co-authored-by: Claude Refactor QuarkOCP_MX (the Quark OCP MX4/MX6 linear scheme) to use the shared vllm/model_executor/kernels/linear kernel abstraction for aiter backend, mirroring the pattern already used by QuarkW8A8Fp8. Previously QuarkOCP_MX hard-coded directly registered/dispatched the AITER gemm_with_dynamic_quant custom op inline, not in line with the kernel-selection framework already pr…

  • 73af7a3 #44513 — [XPU] Add online fp8 quantization test (#44513)
    • 作者: Yan Ma | +28/-7 | 5 个文件

    vLLM adopted new fp8 online frontend by PR #38318 and #40152, which provides more quantization configs including fp8_per_tensor, fp8_per_block and mxfp8. XPU platform supports all of these configs as verified. This PR adds the test for XPU platform. | Model | fp8_per_tensor | fp8_per_block | mxfp8 | Notes | |—|—|—|—|—| | Dense model | Y | Y | Y | | | MoE model | Y | Y | Y | For pure per …

  • 02b6ecf #44527 — [ROCm][DSv3.2] Eliminate per-decode FillFunctor launches in sparse-MLA hot loop (#44527)
    • 作者: frida-andersson | +0/-2 | 1 个文件

    On the ROCm sparse-MLA decode path, two FillFunctor kernels fire on every indexer-bearing decode step with no observable output — pure launch + bandwidth overhead: 1. FillFunctor — out_logits.fill_(float("-inf")) in rocm_fp8_paged_mqa_logits. Upstream added a persistent current_workspace_manager() workspace for out_logits (pre-filled once on allocation), but left the per-call fill_ in pl…

  • 1206891 #47764 — [ROCm][KVConnector][MoRI-IO] Fix WRITE-mode remote-TP rank collapse (#46332 follow-up) (#47764)
    • 作者: avininjamay8 | +15/-2 | 2 个文件

    In WRITE mode, each prefill TP rank must target the correct decode TP rank for KV transfer. #46332 introduced heterogeneous TP support, requiring the prefill to query the decode’s TP size (remote_tp_size) from the proxy to compute the correct target rank. The rank mapping reads peer TP size from kv_transfer_params, but the proxy sets remote_tp_size not tp_size. The old code read tp_size which was …

  • 60b3d39 #50057 — [Docs] Remove experimental warning for EP (#50057)
    • 作者: Woosuk Kwon | +0/-3 | 1 个文件
  • 60417b4 #50034 — [Core][PCP] Select MRV2 when PCP is enabled (#50034)
    • 作者: Lucas Wilkinson | +9/-0 | 1 个文件
    • select Model Runner V2 automatically when prefill context parallelism is enabled - reject an explicit V1 model-runner override with a clear configuration error Related to #49810. ## AI assistance OpenAI Codex assisted with investigation, implementation, and PR preparation. Reviewed by the submitter.
  • 272abd5 #47920 — [Tests][Spec Decode] Add gemma4 MTP acceptance rates test (#47920)
    • 作者: Giancarlo Delfin | +120/-48 | 4 个文件

    This PR Gemma4 MTP is lacking acceptance tests for both MRV1 and MRV2. I generalized the existing test_dflash_acceptance_rates test into test_acceptance_rates and extended it to test dflash (Qwen/Qwen3-8B + z-lab/Qwen3-8B-DFlash-b16) and gemma4 MTP (google/gemma-4-E4B-it + google/gemma-4-E4B-it-assistant). The test iterates over the specified datasets in expected_acceptance_lengths, runs the eva…

  • 1e34a13 #49096 — Fix Humming non-gated MoE (#49096)
    • 作者: Netanel Haber | +111/-7 | 3 个文件

    Enables Humming MoE for NemotronH models with non-gated squared-ReLU experts. Humming previously assumed every w13 contained two gated projections, producing incorrect weight and buffer shapes. ### Results for nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 | Backend | Output tok/s | TTFT | GSM8K | |————|————-:|————-:|——————–| | Marlin | 4266.40 | 21,750.91 ms | 12…

  • ebcef33 #49987 — Fix MQA with tensor parallelism on transformers modeling backend (#49987)
    • 作者: Miłosz Grunwald | +396/-44 | 7 个文件

    Fix k and v replication for Multi-query attention on tp>1 by adding a fuser, to enable models like bigcode/starcoder to run with transformers modeling backend on multi gpu setup. This problem was caused by migration of starcoder to transformers backend in this pr: #30966 vllm serve bigcode/starcoder –dtype bfloat16 –tensor-parallel-size 2 –max-model-len 8192 –gpu-memory-utilization 0.85 –port…

  • 1053e24 #46765 — [ROCm][Quantization][5/N] Refactor quark_moe w8a8-int8 w/ oracle (#46765)
    • 作者: amd-sourjya | +234/-17 | 8 个文件

    Continues the ROCm Quark MoE oracle refactor series (#39136 w4a8, #41436 w4a4, #43721 fp8) for the w8a8-int8 path. - Refactor QuarkW8A8Int8MoEMethod to select its experts kernel via select_int8_moe_backend and dispatch through the modular FusedMoEKernel (honoring –moe-backend), instead of calling fused_experts directly. - Enable INT8 in TritonExperts._supports_quant_scheme on ROCm CDNA GPUs (…

  • b2f9e4c #50004 — [DSv4 Perf] Adaptive topk width, 1.0% E2E throughput improvement (#50004)
    • 作者: Wentao Ye | +56/-7 | 2 个文件

    Let’s assume token num = 2 Originally: Always use maximum length Now based on current context ### Acc ### Perf

  • 831d384 #48879 — [Core] Fail fast when /dev/shm is too small for the shm ring buffer (#48879)
    • 作者: Andrea Tassi | +72/-1 | 2 个文件

    [Core] Fail fast when /dev/shm is too small for the shared-memory ring buffer With tensor parallelism (TP ≥ 2), MessageQueue allocates a POSIX shared-memory ring buffer (default max_chunk_bytes=24 MiB × max_chunks=10 ≈ 240 MiB) in /dev/shm. tmpfs allocation is lazy: SharedMemory(create=True, size=…) succeeds even when /dev/shm is smaller than the requested buffer, and physical pages only f…

  • 99de48e #49982 — Fix MLA padding and grouped topk routing in the Transformers modelling backend (#49982)
    • 作者: Harry Mellor | +227/-39 | 5 个文件
    • Anchor the router’s scorer on the last topk in the gate’s graph — DeepSeek’s group-limited routers topk over expert groups first, so the scorer was sought upstream of the wrong node and the block was declined. - Allow e_score_correction_bias in the gate’s state (DeepSeek-V3 noaux) and carry it onto the rebuilt gate, rather than rejecting any extra state. - Pass use_grouped_topk, num_expert_gro…
  • bf2b45b #42669 — [Attention] Integrate FlashAttention 4 SM100 headdim 256 support (#42669)
    • 作者: Matthew Bonanni | +66/-34 | 7 个文件

    FA4 added support for headdim 256 in https://github.com/Dao-AILab/flash-attention/pull/2412 (and optimized it in https://github.com/Dao-AILab/flash-attention/pull/2487 and https://github.com/Dao-AILab/flash-attention/pull/2488). This support was integrated into vLLM with #41052. This PR is a follow-up to activate the headdim 256 support. > [!NOTE] > This kernel is currently slower than TRTLLM_RAGG…

  • 8112b6c #49364 — [MRV2] Always build attn metadata at capture time (#49364) (#49995)
    • 作者: Nick Hill | +35/-21 | 2 个文件
  • e3c2fc3 #49491 — [Rust Frontend][gRPC] Add server and model discovery (#49491)
    • 作者: Connor Carpenter | +615/-269 | 18 个文件

    Add GetServerInfo and GetModelInfo to the Rust frontend’s gRPC control API. This PR includes the two service-separation commits from #49466 because that dependency is not yet in main: - Rename the wire inference service from vllm.Generate to vllm.Inference. - Split inference and control protobuf definitions and Rust implementations. - Keep generated exports and service registration in grpc/mod.rs….

  • d2ca300 #47711 — [MRV2][Performance] Skip no-op FP32 logits materialization (#47711)
    • 作者: Song Zhixin | +21/-0 | 1 个文件

    MRV2 currently materializes the logits tensor as FP32 before applying sampling parameters, even when every active request is a no-op: temperature is 0 or 1, top-k equals the vocabulary size, top-p is 1, min-p is 0, and there are no penalties, logit biases, or bad words. The Gumbel sampling kernel already converts loaded logits to FP32 in registers. This PR adds a no-op fast path that returns the o…

  • 59a6b04 #49204 — [Core] Fix internal LB load-balancing (#49204)
    • 作者: Nick Hill | +188/-59 | 7 个文件

    Previous PR https://github.com/vllm-project/vllm/pull/30739 added support for efficient DP without EP. The coordinator was kept to propagate engine queue stats for loadbalancing, but there was an omission whereby these stats were not actually getting published, which could result in significant imbalance between the ranks. As well as fixing this (by also calling _maybe_publish_request_counts in th…

  • dbccc5a #48912 — [Model] Enable EVS for Qwen3.5 (#48912)
    • 作者: Rui “Garry” Gao | +32/-12 | 1 个文件

    Add Qwen3.5 support for Efficient Video Sampling (EVS), which was previously disabled in a hard-coded fashion. Per the test results below, on Qwen3.5 EVS works, with characteristics (accuracy decay, TTFT improvement) comparable to Qwen3-VL. Test accuracy decay with –video-pruning-rate 0.75 on Qwen3.5-9B model on Video Bench (subsets MSRVTT+ActivityNet+Driving-decision-making). Test TTFT on random…

  • 92e8518 #49957 — Improve Transformers modelling backend fx tracer (#49957)
    • 作者: Harry Mellor | +307/-128 | 9 个文件
    • Trace with meta tensors: every traced op also runs on meta tensors, so len, iteration and .shape unpacks come from PyTorch’s meta kernels instead of the hand-written rules in _infer_len/_rank. - Record untraceable attention interfaces as opaque leaf calls (flagged in the node’s meta, with the declared return length). vllm_attention_forward used to end the trace early; the graph now reaches the m…
  • 77cba02 #48123 — [KV Offloading] Per-request tier filtering with TierFilter/TierMatcher (#48123)
    • 作者: Ronen Schaffer | +301/-52 | 16 个文件

    Adds per-request tier filtering to KV cache offloading. Requests can specify which secondary tiers to load from via kv_transfer_params[“kv_load_tiers”], filtering by medium and locality. See detailed design discussion: https://github.com/vllm-project/vllm/pull/48123#issuecomment-4979058299 Key changes: - Medium enum: CPU/STORAGE (coarse granularity, both FS and OBJ → STORAGE) - TierMatcher(NamedTu…

  • 30fbd05 #49909 — [ROCm] Use backend-default dot precision for ReplaySSM (#49909)
    • 作者: Andreas Karatzas | +33/-21 | 2 个文件
    • Let ROCm Triton select its native dot precision instead of passing the unsupported CUDA-only tf32x3 value. - Keep tf32x3 on CUDA and move the ROCm Mamba kernel group from MI250 to MI300 so the default ROCm path is exercised.
  • 0906123 #48841 — [ROCm] [Model] Enable TML inkling (#48841)
    • 作者: TJian | +9213/-9 | 33 个文件

    This is the first enablement PR for inkling model on ROCm. This PR also targets the more efficient variant of the model which is mxfp4. There are possible path for loading and use nvfp4 weight, however, we could only support it through inefficient emulation code path. The weight is generated and distributed by https://huggingface.co/lightseekorg/Inkling-MXFP4 instead. There is a continuous effort …

  • cbc3a87 #49907 — [Tokenizer] Use HF config for HF tokenizers (#49907)
    • 作者: Andreas Karatzas | +10/-4 | 1 个文件
    • Resolve the tokenizer backend before loading config and request HF config for HF-backed tokenizers. - Keep native Mistral auto-detection while preventing dual-format repositories from selecting the wrong tokenizer class.

🐛 Bug Fix

  • 30217b0 #49877 — [Bugfix][KV Offload][P2P] Scope serve state to fetch rounds (#49877)
    • 作者: Itay Etelis | +573/-393 | 6 个文件

    FIX #49820 The server keeps a single outbound slot per kv_request_id, but a request can go through several lookup/fetch rounds as its prefix grows. When rounds overlap, finishing one round wipes the supply that was already pinned for the next one, so the next fetch either stalls for the full 30s load timeout (the trace in the issue) or trips the duplicate-fetch guard and takes down the whole sessi…

  • 1e81853 #49964 — [Bugfix][KV Offload] Keep Mamba block span unscaled under DCP (#49964)
    • 作者: Jonguk Cheong | +66/-2 | 2 个文件

    Native KV offloading currently multiplies every cache group’s token span by the decode context parallelism world size. This is correct for attention KV, which is sharded across DCP ranks, but Mamba state is replicated and keeps its original block size. With 16 token attention and Mamba blocks under DCP=2, the current configuration derives 32 tokens per block for both groups. The correct spans are …

  • b6cbba8 #48391 — [Bugfix][Kernel] Fix batch invariance in RMSNorm kernels by pinning block size (#48391)
    • 作者: oops-oom | +189/-21 | 6 个文件

    Fix a batch-invariance gap in the fused_add_rms_norm CUDA kernel. The kernel selects its block size from the token count: max_block_size = (num_tokens < 256) ? 1024 : 256, independent of whether batch-invariant mode is enabled. Under VLLM_BATCH_INVARIANT=1, the residual RMSNorm path (RMSNorm.forward_cuda with a residual → ops.fused_add_rms_norm) routes to this kernel. As a result the same token …

  • 62d8db7 #50131 — [Bugfix] Add missing vllm/models/kimi_k3/__init__.py (#50131)
    • 作者: Harry Mellor | +4/-0 | 2 个文件

    #50090 added vllm/models/kimi_k3/ but never added vllm/models/kimi_k3/init.py, so the directory is an implicit namespace package. This broke the docs build on main: Failing build: https://app.readthedocs.org/projects/vllm/builds/33792546/ (at 61ac368). api-autonav walks the source tree and finds the subpackages, but Griffe cannot resolve vllm.models.kimi_k3.amd because its parent has no __in…

  • 948107a #48589 — [Bugfix] Enhance extra_config handling for layer name suffix matching (#48589)
    • 作者: Xin He | +57/-0 | 2 个文件

    This pull request improves how the configuration is selected for layers with extra configuration by adding support for suffix matching. This addresses situations where the keys in extra_config are short names (like “lm_head”), but the actual layer_name is fully qualified (like “model.language_model.lm_head”), which can happen due to model nesting. Configuration handling improvements: * Added s…

  • b09688a #49774 — [Bugfix][Spec Decode] Preserve draft buffers across level-2 sleep (#49774)
    • 作者: aoshen02 | +12/-11 | 1 个文件

    Fixes two problems in the level-2 sleep/wake buffer-restore path, and removes a mechanism that the first fix makes redundant. 1. Speculative draft-model buffers were never snapshotted. Level-2 sleep releases the weights pool outright (offload_tags=()), so on resume the pages are remapped but uninitialized. Registered buffers are not recoverable from the checkpoint (cos_sin_cache and friends ar…

  • 03a2d03 #49966 — [Bugfix] Respect cgroup memory limits on all platforms (#49966)
    • 作者: Chauncey | +1/-3 | 1 个文件

    [Bugfix] Respect cgroup memory limits on all platforms When starting vLLM inside a Pod, vLLM incorrectly detects the available memory size. I configured the memory limit, but vLLM does not read it correctly. ## Test Result —

  • 90245f4 #50073 — [Bugfix] Fix multi-modal support on CPU MRV2 (#50073)
    • 作者: Li, Jiang | +18/-1 | 3 个文件

    Patch several unsupported operations of multi-modal support on CPU MRV2 ## Test Result —

  • f472ab0 #49621 — Remove triton per group quant [ROCm] [Bugfix] (#49621)
    • 作者: afriedri | +8/-78 | 6 个文件

    On ROCm, dynamic FP8 per-group (block-scale) activation quant was routed through a vllm.triton_per_token_group_quant_fp8 custom op whenever the block-scaled linear kernel chose its tuned Triton GEMM path (AiterFp8BlockScaledMMKernel, use_triton=True). That op is a thin wrapper that already dispatches to the C++ _C.per_token_group_fp8_quant kernel on ROCm (#42758) — so it adds no compute, but i…

  • d223c90 #50060 — [Bugfix] Only pad transformers backend value when it is narrower (#50060)
    • 作者: Nick Hill | +6/-3 | 1 个文件

    #49982 added padding of value up to the query/key head size in vllm_attention_forward for expanded MLA, guarded on head_dim_v != head_dim_qk. #49987 then added support for packed grouped/multi-query QKV projections, whose attention modules keep key/value packed rather than split per head, so value.shape[-1] is kv_heads * head_dim - i.e. larger than the query head size whenever kv_heads > 1. The …

  • 52c3c4a #49947 — [Bugfix][KV Offload][OBJ] Preserve job completion during cleanup (#49947)
    • 作者: MINJUN GIL | +133/-7 | 2 个文件

    The OBJ secondary tier removes a transfer from _transfers before releasing its NIXL resources. If release_xfer_handle() raises, no JobResult is published and the OBJ tier has already discarded the only state needed to retry cleanup. The parent TieringOffloadingManager therefore cannot pop the corresponding _transfer_jobs entry or call primary.complete_read(). For a primary-to-OBJ store, the CPU ca…

  • fbb1ef6 #49634 — [Bugfix] Fix DeepseekV4FP8 Quark MXFP4 crash on list-valued weight (#49634)
    • 作者: Colin Z | +5/-1 | 1 个文件

    Fix an AttributeError crash in DeepseekV4FP8Config._is_quark_mxfp4_ocp that breaks loading of Quark-quantized checkpoint whose global_quant_config.weight is a list. Per the Quark schema, weight is Optional[Union[QTensorConfig, List[QTensorConfig]]], so it can be a dict, a list (multiple specs, e.g. NVFP4 with 2-level scale), or None. The original code assumed a dict, crashing on the list form:…

  • 28158b2 #48886 — [ROCm] [BugFix] Fix Quark GLM-5.2 Checkpoint inference: indexer wk per-channel FP8 dequant + missing sparse-MLA metadata fields (#48886)
    • 作者: Colin Z | +129/-21 | 5 个文件

    This PR addresses two independent bugs blocking quark quantized GLM-5.2 checkpoints with Attn quantized to PTPC FP8 from running end-to-end on ROCm (MI355X / gfx950). ### Per-channel FP8 scale for fused indexer wk (deepseek_v2.py): _try_load_fp8_indexer_wk dequantizes the FP8 indexer wk weight to BF16 at load time so it can be fused with weights_proj. However, it unconditionally reads scale_inv.sh…

  • ed13deb #49985 — [Bugfix][CPU] Fall back to torch for unaligned swigluoai on NEON/vec MoE (#49985)
    • 作者: oops-oom | +6/-0 | 1 个文件

    I found this while triaging CI failures on my own PR #48391. That PR touches csrc/, so it triggers the csrc/-gated run-all (which includes the Arm CPU Test); the Arm CPU Test came back red, and it turned out to be a pre-existing regression on main, not caused by #48391. PR #49591 added the swigluoai grouped-gemm fallback only on the AMX path (raise-on-unaligned) plus a test asserting non-A…

  • 15d65f8 #41131 — [Bugfix] Changed speech to text chunk timestamp to cumulative approach (#41131)
    • 作者: TobyJBell | +101/-11 | 5 个文件

    Fix related to issue: #32588 The bug occurs when transcribing long audio (> 30s) using vLLM whisper, the timestamps of which become increasingly inaccurate (up to 1s per chunk). This happens because the old approach assumes split_audio cuts exactly at 30s, but actually will search within 1-second window before the expected value. So it could land anywhere within that, 29.2 or 29.8 for example. Thi…

  • 3f47a83 #49846 — [Bugfix] Fix VLLM_ENFORCE_STRICT_TOOL_CALLING mutation in tests (#49846)
    • 作者: yzong-rh | +26/-44 | 3 个文件

    Fix a code smell / latent bug where tests sets env variable VLLM_ENFORCE_STRICT_TOOL_CALLING=0, potentially leading to tests assuming VLLM_ENFORCE_STRICT_TOOL_CALLING=1 to fail. Occurred for https://github.com/vllm-project/vllm/pull/45560. ## Test Result —

  • 27d7061 #49963 — [Bugfix] Restore truncate_prompt_tokens for Jina rerank/score online (#49963)
    • 作者: Umut Polat | +69/-1 | 2 个文件

    Fix a regression where truncate_prompt_tokens (and truncation_side) is silently dropped on the Jina rerank/score online path, so a request that asked to truncate is instead rejected with HTTP 400. JinaRankingIOProcessor.get_request_factory_online swaps ctx.request for a proxy PoolingCompletionRequest and delegates to the base factory, which reads truncation off ctx.request: The proxy was built…

  • ef9975d #45828 — [Bugfix] Reject pipeline parallelism for DiffusionGemma (#45828)
    • 作者: Guan-Ming Chiu | +0/-6 | 1 个文件

    Canvas state is advanced only by the sampler on the last PP rank, but every rank reads it for forward inputs, and the generic PP path broadcasts only token ids. Fail closed at startup instead of crashing in warmup. Thus, would like to remove SupportsPP interface to prevent error in advance - set PP=2 and run with DiffusionGemma ## Test Result - PP=2: fast ValueError, no warmup crash. PP=1: unchang…

  • 81962bb #49043 — [Bugfix]Reject invalid FlashInfer MNNVL workspaces (#49043)
    • 作者: rongfu.leng | +6/-0 | 1 个文件

    Fixs: https://github.com/vllm-project/vllm/issues/49041 On topologies without symmetric-memory multicast support, FlashInfer can construct an MNNVL allreduce workspace whose mc_ptr is null. vLLM treated the workspace as initialized and passed it into fused allreduce kernels, which dereferenced the null multicast mapping and failed with a CUDA illegal memory access. Validate mc_ptr immediately afte…

  • 394beb6 #49843 — [Bugfix][ROCm] Use batch DMA for CPU KV cache loads (#49843)
    • 作者: Andreas Karatzas | +16/-2 | 2 个文件
    • Add ROCm to the existing XPU batch-DMA fallback for CPU-to-GPU KV transfers. - Keep CUDA, XPU, and every other prior dispatch path unchanged. The ROCm Triton path directly loaded a raw shared-mmap host pointer. The reproduced fault address matched the mmap base plus the selected block offset in the first case above the Triton descriptor threshold. Buildkite: V1 Core + KV + Metrics
  • 7f599d7 #46913 — [communication] [bugfix] fix quickreduce acc error in cudagraph mode (#46913)
    • 作者: haoyangli0109 | +154/-8 | 2 个文件

    1. cause: Once flag_color is fixed by graph, it remains unchanged for each round → The written flag value repeats in each round and cannot be distinguished from the residual value of the previous round → The waiting party is prematurely satisfied by the old value and is immediately granted access → At this point, since the data for the current round has not yet been fully transmitted, the syst…

  • eb290ab #49591 — [Bugfix][CPU] Zero-pad MoE intermediate size for grouped-gemm TP alignment (#49591)
    • 作者: Li, Jiang | +211/-14 | 2 个文件

    The CPU AMX/vector grouped-gemm MoE kernels (csrc/cpu/cpu_fused_moe.cpp) tile the expert intermediate dimension in fixed 32-wide blocks with no tail/remainder handling. CPUFusedMOE.check_grouped_gemm only enables the fast kernel when the per-partition moe_intermediate_size (i.e. moe_intermediate_size // tp_size) is a multiple of 32; otherwise it silently falls back to cpu_fused_moe_torch, a per-ex…

  • 8061dc2 #49392 — [Bugfix] Normalize sparse MLA warmup compression ratios (#49392)
    • 作者: Xiaochang Wu | +24/-2 | 2 个文件

    Normalize DeepSeek V4 compression ratios before generating sparse-MLA Triton warmup keys, matching the runtime attention path. DeepSeek V4 configs use 0 to represent uncompressed/SWA-only layers. Runtime attention already converts those values with max(1, ratio), but BuildPrefillChunkMetadataKernel.get_warmup_keys() used the raw config values. This generated a COMPRESS_RATIO=0 specialization conta…

🧪 CI/Tests

  • 94100b5 #49340 — [CI] Wire untethered test files into CI jobs (#49340)
    • 作者: Nick Hill | +88/-13 | 9 个文件

    An audit of .buildkite pytest targets against tests/**/test_*.py found dozens of test files that no CI job ever runs. Wire in the ones that pass on current main (every set validated on B200-class hardware, or CPU-only for CPU tests, before wiring): - tests/kernels root: after the kernels/ subdir categorization (#16799), the root stopped being collected, and 19 files added since then never ran. Add…

  • 25ace8f #49881 — [CI] Increase Qwen3.5 MTP GSM8K generation length (#49881)
    • 作者: Jiangyun Zhu | +2/-1 | 1 个文件

    The Qwen3.5 MTP3 GSM8K evaluation is frequently truncated by the default max_tokens=256. The evaluator extracts the last integer from the generated text even when the reasoning is incomplete, so a truncated chain of thought can be scored as a valid but incorrect answer. This caused the MTP configuration to fail in Buildkite build #80270: - Qwen3.5-397B-A17B-NVFP4-DEP2-MTP: 0.8469 accuracy, 5.8% in…

  • 247470f #48164 — [CI] Add PyTorch stable ABI audit check (#48164)
    • 作者: Chris Leonard | +130/-1 | 6 个文件
    • Add a new Buildkite step, Torch Stable ABI Audit, that runs after the CUDA CI image build and uses torch-abi-audit to verify vLLM’s compiled extensions comply with the PyTorch stable ABI. - Fail CI if any extension links unstable libtorch symbols (at:: / c10:: / etc.) unless it is listed in ALLOWED_UNSTABLE_LIBRARIES in .buildkite/check-torch-abi.py. - Fail CI if an allowlist entry is stale …
  • 88402a4 #49945 — [Test] Skip ROCm AITER MLA prefill tests on non-ROCm platforms (#49945)
    • 作者: Liangliang Ma | +6/-0 | 1 个文件

    TestROCmAiterFAPrefillSelection exercises ROCm-specific AITER FlashAttention MLA prefill backend gating. On non-ROCm platforms (e.g. XPU) patching vllm.platforms.rocm forces its module-level import, which runs torch.cuda.get_device_properties and fails when torch is not built with CUDA. Guard the class with a ROCm-only skipif.

  • 99b57a4 #50086 — [CI][ROCm] Soft fail LoRA mirror (#50086)
    • 作者: Andreas Karatzas | +1/-0 | 1 个文件

    Soft fail LoRA mirror until proper fix on ROCm.

  • e68bfc2 #50041 — [CI][ROCm] Soft-fail Python-only installation mirror (#50041)
    • 作者: Andreas Karatzas | +1/-0 | 1 个文件

    Soft failing the Python-only installation mirror to regate fast.

  • 53f6dd5 #49690 — [CI][ROCm] Fix test_ocp_mx_wikitext_correctness reference value (#49690)
    • 作者: fxmarty-amd | +1/-1 | 1 个文件

    “tests/quantization/test_quark.py::test_ocp_mx_wikitext_correctness[tp_size:1-config:AccuracyTestConfig(model_name=‘fxmarty/qwen_1.5-moe-a2.7b-mxfp4’, excepted_value=12.4)]” has been failing for some time in AMD CI, see e.g. https://buildkite.com/vllm/amd-ci/builds/11168/list?sid=019f8e34-4da4-4a99-b00b-7effcc173a91&tab=output The cause is https://github.com/vllm-project/vllm/pull/46643 the change…

  • 99115fc #49912 — [CI] Initialize DeepEP FP8 test weights (#49912)
    • 作者: Andreas Karatzas | +8/-2 | 1 个文件
    • Replace allocator-dependent FP8 source weights with small bounded random values. - Allocate through current_platform.device_type while preserving the buffer-reuse and tolerance fixes from #46758.
  • fd10e89 #43559 — [Test] Regression test for hybrid-Mamba eagle cache-peek in Mooncake connector (#43559) (#48361)
    • 作者: Rishi Puri | +39/-0 | 2 个文件
  • bc3629b #49732 — [ROCm][CI] Skip three torchao tests of gfx950 until torchao==0.18 is released (#49732)
    • 作者: fxmarty | +25/-0 | 1 个文件

    Submitting from this account as @fxmarty-amd is getting open pull request limit has been reached for this user —- On AMD MI350/gfx950, test_online_quant_config_dict_json, test_online_quant_config_file, and test_reload_weights in tests/quantization/test_torchao.py use Float8DynamicActivationFloat8WeightConfig, which fails on torchao releases up to v0.17.0 because that check only allows is_sm_at_l…

  • 312ea82 #49837 — [CI][ROCm] Make hf-xet reconstruction safe on shared NFS (#49837)
    • 作者: Andreas Karatzas | +13/-0 | 1 个文件
    • Keep Xet and the persistent Hub cache enabled while moving ephemeral Xet state to local storage. - Disable high-performance vectored reconstruction only when HF_HOME is NFS-backed.
  • afc9452 #49939 — [XPU][CI] Use platform device in InputBatch V2 test (#49939)
    • 作者: liuzhenwei | +2/-1 | 1 个文件

    Fix the InputBatch V2 test to use the current platform device instead of hard-coding CUDA. ## Test Result —

🦀 Rust Frontend

  • 9069a57 #49040 — [Core][Frontend] Add weight version tagging for RL rollouts (#49040)
    • 作者: Shuolei Wang | +142/-18 | 16 个文件

    Part of #48306, this PR implements the engine-level externally managed weight-version metadata and query/update APIs from Section 2.2. It builds on the weight-update lifecycle introduced in #39212. For online RL inference: - Each EngineCore starts with weight_version=“default”. - Callers may pass an opaque string such as “step-42” to finish_weight_update. The supplied version is published only aft…

✨ New Feature

  • 60915c9 #47750 — [Feature] Add VidCom2 video token pruning (#47750)
    • 作者: nvbfalk | +421/-48 | 17 个文件

    Add VidCom2 (Video Compression Commander) as a second training-free video token pruning method alongside EVS, enabled via –video-retention-ratio. - Paper: Liu et al., “Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models”, EMNLP 2025 — https://arxiv.org/abs/2505.14454 - Reference implementation (Apache-2.0): <https://github.com/xuyang-liu16/Vid…

🔧 Refactor

  • b5bcb3c #49745 — [Refactor] Remove dead code in multiple files (#49745)
    • 作者: Wentao Ye | +0/-519 | 22 个文件

    Remove dead code in multiple files

🔩 Misc

  • 2b465b2 #49988 — [Misc][PD] Nixl cleanup get_backend_aware_kv_block_len and virtually_split_kv_in_blocks (#49988)
    • 作者: Nicolò Lucchesi | +44/-85 | 2 个文件

    Following up on https://github.com/vllm-project/vllm/pull/44456/ with some more nixl-specific cleanup. Now that KV are packed and we can safely assume KV regions are registered together, a few things can be simplified. - virtually_split_kv_in_blocks special handling is dead-code, as it resolves to being always False now - we can also bid get_backend_aware_kv_block_len farewell, as the clumsy logic…

⚡ Performance

  • 04502de #49607 — [Perf] Hash videos by source bytes (#49607)
    • 作者: Guan-Ming Chiu | +94/-21 | 8 个文件
    • Video requests hash the full decoded frame buffer; images already hash their compressed source - Hash the smaller of source bytes and decoded frames, plus the sampling metadata - Byte-count comparison, so it is hardware-independent and always picks the cheaper buffer - Follow-up: decode options absent from the metadata still stay out of the key, like images - pytest tests/multimodal/test_parse.p…
  • 56c96b0 #48774 — [Perf] Tune LL BF16 Router GEMM (#48774)
    • 作者: Roberto L. Castro | +93/-20 | 2 个文件

    This PR tunes the ll_bf16_gemm router GEMM dispatch heuristic for critical models: GLM5.2, DSv3.2, DSv4-Flash, and DSv4-Pro on Hopper and Blackwell GPUs. Router GEMM has shown up as an expensive kernel if it is not treated carefully. For the most critical models, we have historically used ad-hoc, PTX-based kernels focused on LL scenarios. However, those kernels are usually limited to a single shap…

  • a89015c #48739 — [Perf] Make merge attention context count a runtime argument (#48739)
    • 作者: liminfei-amd | +2/-2 | 1 个文件

    Makes merge_attn_states_kernel’s batch-varying prefill_tokens_with_context a runtime argument instead of a tl.constexpr, preventing a new Triton specialization for each distinct value. Partial fix for #48650. ## Why this is not a duplicate Rechecked on 2026-07-15 immediately before preparing this draft: - #48650 is open, unassigned, with no comments. - PR #48734 now implements the issue’s count_ex…

  • 96fa3f4 #49659 — [Perf] Skip ll_bf16 router GEMM warmup for non-MoE models (#49659)
    • 作者: neweyes | +1/-1 | 1 个文件

    Skip unnecessary ll_bf16 router GEMM warmup for non-MoE models. _warmup_ll_bf16_router_gemm() initializes router GEMM kernels that are only used by MoE models. Previously, the warmup was executed on all Hopper/Blackwell GPUs regardless of model type, introducing unnecessary startup overhead for dense models. This change adds an is_moe check so that router GEMM warmup only runs when the loaded mode…