共 86 个 commit，涉及 442 个文件，+16611/-5851 行变动。

概要

统计项 数值
Commit 数 86
变更文件 442
新增行数 +16611
删除行数 -5851

Commit 列表

🖥️ Kernel

  • 2ec6594 #36902 — [Kernel][Helion][1/N] Add Helion kernel for per_token_group_fp8_quant (#36902)
    • 作者: Xiaohong (Sean) Chen | +4347/-4 | 10 个文件

    This PR is to add Helion kernel for operation. It follows the implementation from the vllm c version. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. Autotuning and benchmarking in H200 and B200 were contributed by @gmagogsfm. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: a…

📦 Other

  • 79f8c5b #42331 — [Metrics] Scope unregister_vllm_metrics() to strictly “vllm:” metrics (#42331)
    • 作者: vraiti | +1/-1 | 1 个文件

    unregister_vllm_metrics() currently uses “vllm” in collector._name to decide which collectors to remove from the Prometheus registry. This is a substring match that removes every collector whose internal name contains vllm anywhere, including metrics registered by other subsystems or downstream extensions that happen to use vllm in their metric names. This came up when trying to add Prometheus s…

  • f81daf8 #41797 — [Attention] add triton diff-kv backend for mimo (#41797)
    • 作者: Jiangyun Zhu | +1041/-9 | 8 个文件

    Fix https://github.com/vllm-project/vllm/issues/41519 ## Test Result FA |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|——:|—————-|—–:|———–|—|—–:|—|—–:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9378|± |0.0067| | | |strict-match | 5|exact_match|↑ |0.9371|± |0.0067| triton |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|—-…

  • 4085ff7 #44594 — [Core] Add kvcache watermark to reduce preemptions (#44594)
    • 作者: Nick Hill | +291/-8 | 7 个文件

    Configurable percentage of total kvcache blocks to reserve as buffer when allocating new request blocks. Disabled by default. Does not apply to first new request when cache is empty. #### Benchmark Config - Model / HW: Qwen/Qwen2.5-7B-Instruct, TP=1, 1× NVIDIA GB200 - KV cache: 16 GiB → 299,584 tokens (near-critical: ~1.5× mean demand), prefix caching off - Engine: –max-model-len 8192…

  • 5edf7ff #45179 — [Core] Release cached device memory under pressure on UMA GPUs during weight loading (#45179)
    • 作者: Michael Goin | +42/-0 | 2 个文件

    Alternative to #44502. Weight repacking in process_weights_after_loading leaves freed checkpoint-format tensors in the caching allocator; on integrated (UMA) devices like DGX Spark the reserved cache starves the OS, which can thrash before an OOM retry would release it. Differences from #44502: - One call site in the loader loop covers all quant methods, not just NVFP4 Marlin - No per-layer gc.col…

  • b78fc47 #45218 — [Docs] Add redirect for moved lmcache examples page (#45218)
    • 作者: Natalie Lin | +1/-0 | 1 个文件

    Add a missing redirect for the examples/others/lmcache path, which was moved to examples/disaggregated/lmcache in #41082 but without a corresponding redirect entry in mkdocs.yaml. This causes a broken link for users arriving from search engines: the top Google result for “vllm lmcache” is the v0.10.1 docs page, whose “latest stable version” banner links to stable/examples/others/lmcache.html, whic…

  • 03878d1 #44992 — Deprecations for v0.23 and v0.24 (#44992)
    • 作者: Harry Mellor | +102/-676 | 35 个文件

    Perform deletions for deprecations scheduled for: - v0.23 - these should technically have already been deleted as v0.23 has already been cut - v0.24* - this will be the next minor release to be cut *this PR does not include the deletion of the Transformers v4 code path. This is somewhat more complicated and will be done in a follow up PR.

  • 55911db #44243 — [PD][Core] Fix Mamba prefix cache hit rate in PD disaggregation (#44243)
    • 作者: zhanqiuhu | +534/-3 | 5 个文件

    Co-authored with @underfituu. Fix the bug described in https://github.com/vllm-project/vllm/pull/42524; overwrite find_longest_cache_hit to bypass truncation of full attention groups. Note on the expected behavior: the last state of Mamba will always be transfer, but full attention will only transfer the prefix cache miss part.

  • cc640ee #45030 — [Rust Frontend][Metrics] Export vllm:lora_requests_info from frontend (#45030)
    • 作者: Will Eaton | +383/-36 | 8 个文件

    Add support for reporting LoRA adapter metrics to the rust frontend, this is required for llm-d compatibility with the lora-aware-routing affinity scorer. ref: https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/scheduling/scorer/loraaffinity The engine’s SchedulerStats never carries adapter names: in the python frontend they are computed frontend-side by LoRARequestStates in…

  • ebc6ef9 #43805 — Hidden states extraction improvements (#43805)
    • 作者: Fynn Schmitt-Ulms | +457/-236 | 7 个文件

    The current ExampleHiddenStatesConnector saves the hidden states in wait_for_save method of the KVConnector. The challenge with this, is that when running with chunked prefill, these method is called multiple times per request. The current solution is to just disable chunked prefill and enforce that when using the connector. This pr removes this requirement, by moving the save step into get_finish…

  • ab3a1fd #45244 — minicpmv4_6: fix ImageSize (W,H) order for placeholder token calculation (#45244)
    • 作者: tc-mb | +8/-81 | 2 个文件

    Fix a bug where _compute_visual_tokens passes ImageSize(width, height) directly to transformers image processor methods (get_sliced_grid, find_best_resize, get_refine_size) that expect (height, width) order. For certain aspect ratios (e.g. 365x211) this causes the text-side placeholder count to mismatch the actual vision embedding count, triggering a ValueError crash. - minicpmv4_6.py: convert Ima…

  • c3662b3 #44733 — [KV offload] Parallel-agnostic fs-tier cache for single full-attention group (#44733)
    • 作者: Itay Etelis | +102/-3 | 4 个文件

    Tests pytest tests/v1/kv_offload/tiering/test_fs_tier.py — adds 3 predicate cases (single full-attn → agnostic; multi-group → off; non-full-attn → off). Validated end-to-end on 4×H100 (store TP=2 → load TP=4, same cache dir): | Model | Result | |—|—| | Qwen2.5-7B (full attention) | 400/403 tokens loaded from the TP=2 cache; output identical to fresh TP=4 | | DeepSeek-V2-Lite (MLA) | Short r…

  • e62d00a #45253 — docs: add fix disclosure policy to SECURITY.md (#45253)
    • 作者: Juan Pérez de Algaba | +9/-0 | 1 个文件

    Document that MODERATE and LOW severity fixes are developed as public pull requests, while CRITICAL and HIGH severity issues use a private security fork. The vulnerability management team retains discretion to adjust the approach on a case-by-case basis.

  • 1f60771 #42679 — fix: guard flash-attn rotary import (#42679)
    • 作者: Yufeng He | +7/-5 | 1 个文件

    Fixes #42675. - import flash_attn.ops.triton.rotary directly when vLLM is not on CPU - treat a missing rotary module as unavailable instead of crashing during ApplyRotaryEmb construction - keep the existing fallback path when FA4 no longer provides the old rotary module FA4 can leave the flash_attn root package importable while moving or removing flash_attn.ops.triton.rotary. Checking only the roo…

  • 05d9848 #44923 — [Build] Upgrade CUDA Dockerfiles from GCC 10 to GCC 12 for C++20 compatibility (#44923)
    • 作者: Richard Barnes | +27/-9 | 4 个文件

    Why this is not a duplicate Checked open PRs on 2026-06-08 with gh pr list –search “gcc-10 gcc-12” and –search “c++20 compiler” — no existing PR addresses this. PyTorch PR #167929 raised the minimum C++ standard to C++20 and the minimum GCC version to 11.3. vLLM’s main CUDA Dockerfile and nightly-torch Dockerfile both pin GCC to version 10 to suppress spurious -Wredundant-move warning spa…

  • ef67071 #44783 — [Build] Skip spinloop extension on Python < 3.11 (#44783)
    • 作者: jasen | +17/-13 | 2 个文件

    The spinloop extension uses Py_buffer, PyBuffer_Release, and the “y*” format code in PyArg_ParseTupleAndKeywords — none of which are available in the Python Limited/Stable API. Combined with USE_SABI 3.11, this causes compilation to fail on Python 3.10 because the Limited API headers do not declare these symbols. Guard the spinloop target in both CMakeLists.txt (Python_VERSION check) and setup.py …

  • 432905d #45262 — Only enable PR docs builds manually (#45262)
    • 作者: Harry Mellor | +20/-6 | 1 个文件

    Even with the gating we added recently, queue times are still very long. This PR changes the gate to be fully manual based on the presence of a build-docs label on the PR.

  • 9492362 #45119 — [Security] Apply sanitize_message to Anthropic and STT error paths (#45119)
    • 作者: Juan Pérez de Algaba | +91/-5 | 4 个文件
    • Apply sanitize_message() to all 5 error response sites in the Anthropic API router, its SSE streaming converter, and the speech-to-text WebSocket paths that were missed by the original CVE-2026-22778 fix (PRs #31987, #32319). - These paths catch exceptions in-route and return str(e) directly, bypassing the global FastAPI exception handler that already applies sanitization. PIL UnidentifiedImageE…
  • 7852e50 #43724 — [docs] Document –scheduler-cls base class requirement (extend AsyncScheduler, not Scheduler) (#43724)
    • 作者: Georgii Kliukovkin | +6/-5 | 1 个文件

    Documents the requirement that custom scheduler plugins loaded via –scheduler-cls should extend AsyncScheduler, not Scheduler. The default V1 scheduler is AsyncScheduler; subclassing Scheduler instead disables async scheduling overlap with GPU execution and causes significant latency regression (we measured ~78% on multi-turn workloads before catching the mismatch). ## Context This footgun was id…

  • 0d657e4 #45155 — [Rust Frontend] Fix DeepSeek V3.2 continue_final_message rendering (#45155)
    • 作者: Reid | +22/-3 | 2 个文件

    Fix DeepSeek V3.2 Rust chat rendering for continue_final_message=true. Before this change, the DeepSeek V3.2 renderer ignored the final-assistant continuation mode and always passed prefix=false when rendering assistant messages. As a result, a final assistant prefill could be rendered as a completed assistant message with <｜end▁of▁sentence｜>, instead of remaining open-ended for the model to conti…

  • aa1df36 #44980 — Fix/minicpmv46 missing version (#44980)
    • 作者: 王金旭 | +32/-2 | 3 个文件

    Reported in OpenBMB/MiniCPM-V#1113 (vllm serve openbmb/MiniCPM-V-4_6). ## Fix Two layers: 1. MiniCPMV4_6ProcessingInfo.get_hf_processor() now returns the native MiniCPMV4_6Processor directly instead of the vendored wrapper. MiniCPM-V 4.6 already implements its own image/video handling and prompt-update logic, so it neither needs nor is compatible with the vendored processor designed for 2.x/4….

  • f06aefb #44523 — [CPU] Add missing scalar fallback for CPU W4A8 INT4 GEMM (#44523)
    • 作者: wcy | +61/-15 | 4 个文件

    Summary This PR fills in the scalar fallback implementation for the CPU W4A8 INT4 GEMM path. The scalar path is mainly intended as a correctness/reference fallback when optimized paths such as AVX512/brgemm are unavailable. As part of this, the INT4 W4A8 CPU ops are registered on RISC-V so the direct C++/Torch op path can be exercised there. Before this change, non-AVX targets that reached the…

  • d598d23 #45116 — [Security] Reject non-finite temperature and repetition_penalty values (#45116)
    • 作者: Juan Pérez de Algaba | +63/-0 | 2 个文件
    • Add math.isfinite() validation for temperature and repetition_penalty in SamplingParams._verify_args(). - NaN and Infinity bypass Python’s comparison operators (<, >) due to IEEE 754 float semantics, allowing them to propagate to GPU sampling kernels where they cause undefined behavior or CUDA crashes. - Addresses advisory GHSA-7h4p-rffg-7823.
  • f219788 #44971 — [Security] Fix info disclosure via int32 truncation in GGUF dequantize kernels (#44971)
    • 作者: Juan Pérez de Algaba | +33/-29 | 3 个文件

    Fix an information disclosure vulnerability caused by integer truncation in GGUF dequantize kernels (csrc/libtorch_stable/quantization/gguf/). The to_cuda_ggml_t function pointer typedef declares its element count parameter k as int (32-bit). When a GGUF model has weight tensor dimensions whose product exceeds INT_MAX (e.g. a 65536x65536 matrix), the int64_t product m * n is silently truncated to …

  • 6e64c1b #44565 — [10c/n] Migrate MoE kernels to torch stable ABI (#44565)
    • 作者: Chris Leonard | +1300/-1102 | 34 个文件

    This PR continues the libtorch stable ABI migration (see #26946) for vLLM MoE CUDA kernels by introducing _moe_C_stable_libtorch and moving all of the MoE ops (topk, align, permute/unpermute, grouped topk, and related headers) into csrc/libtorch_stable/moe/. Note: started using the [10x/n] label to indicate that they could be merged in any order (theoretically, there could still be merge conflicts…

  • 2f2c5cf #45236 — [release] Always block release images to dockerhub (#45236)
    • 作者: Kevin H. Luu | +0/-1 | 1 个文件
  • 40e065e #45204 — [Docker] Fix CUTLASS DSL cu13 install order in Dockerfile (#45204)
    • 作者: Mohammad Miadh Angkad | +35/-0 | 1 个文件

    Fix CUDA 13 Docker builds by force-reinstalling nvidia-cutlass-dsl-libs-cu13 last, avoiding uv install-order races with overlapping CUTLASS DSL sibling wheels

  • 0b995f8 #45089 — Use std::bit_cast for type punning in CPU kernels (#45089)
    • 作者: Yuanyuan Chen | +36/-56 | 3 个文件

    Replace std::memcpy-based type punning with std::bit_cast (C++20) in the RISC-V, VXE, and float_convert CPU helpers, adding the necessary and includes. ## Test Result — - [X] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.

  • 43914dd #44624 — [Rust Frontend] Add Python bridge for Rust tool parsers (#44624)
    • 作者: Bugen Zhao | +1302/-38 | 16 个文件

    This PR adds a generic PyO3 bridge that exposes Rust tool parsers to Python through vllm._rust_tool_parser, along with a Python RustToolParser adapter for the existing vLLM ToolParser interface. This is not yet wired into any production parser path. The intent is to provide a future extension point where Python adapters can delegate parser state and grammar to Rust while keeping protocol adaptatio…

  • 3501324 #44942 — [Build] fix self-contradictory precompiled-flag orthogonality test (#44942)
    • 作者: Prajjwal Chittori | +21/-2 | 2 个文件

    tests/test_envs.py::test_precompiled_install_flags_are_orthogonal is self-contradictory and always fails when actually run. It seeds the environment with VLLM_PRECOMPILED_WHEEL_LOCATION (which has enabled VLLM_USE_PRECOMPILED since #22106) and then asserts VLLM_USE_PRECOMPILED() is False — an assertion that can never hold. The bug shipped unnoticed because tests/test_envs.py is not run by any CI l…

  • f272dfd #44774 — [KV Connector] Mooncake store: prefix-cache retention interval for sparse attention (#44774)
    • 作者: Yifan Qiao | +92/-37 | 4 个文件

    Follow-up of #43447. Adds prefix-cache retention support to the Mooncake KV store’s store_mask, so the store writes exactly the blocks a future local prefix-cache hit can actually consume — and no more. Previously store_mask computed a per-lcm-region template (via a dummy-hash pass through find_longest_cache_hit) and tiled it across the request. This refactor instead reuses the engine’s own Single…

  • f31bc2e #44478 — [CPU][RISC-V] Enable oneDNN W8A8 INT8 to run on RISC-V (#44478)
    • 作者: velonica0 | +42/-5 | 4 个文件

    Problem vLLM’s CPU W8A8 dispatcher (vllm/model_executor/kernels/linear/scaled_mm/cpu.py:40-55) has two branches: an x86-only SGL fast path, and an oneDNN path for everyone else. On RISC-V the else is unconditional. But oneDNN itself isn’t compiled in on RISC-V: cmake/cpu_extension.cmake doesn’t gate it on, and csrc/cpu/torch_bindings.cpp doesn’t register the ops behind __riscv_v. Result: loadi…

  • 5d5591d #44887 — [Rust Frontend] Populate cached_token_count in responses (#44887)
    • 作者: Bugen Zhao | +556/-262 | 34 个文件

    Signed-off-by: Bugen Zhao i@bugenzhao.com PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED. This PR currently bases on #44884. Close #44824. Populate cached_token_count in prompt_tokens_details for OpenAI-compatible responses from prefill stats, when –enable-prompt-tokens-details is specified. Also extract TokenUsage struct and ApiServerOp…

  • 85a0ffa #45194 — [CI Bug] Remove qwen test ValueError: No example model defined for Qwen/Qwen-7B-Chat (#45194)
    • 作者: Wentao Ye | +0/-5 | 2 个文件

    Fixes https://buildkite.com/vllm/ci/builds/71234#019eb2da-8fd7-4096-b373-2593fb49305b Qwen has been removed in https://github.com/vllm-project/vllm/pull/45131, so we remove the related tests as well.

  • 18d87a8 #45161 — Deprecate Transformers v4 support (#45161)
    • 作者: Harry Mellor | +62/-268 | 19 个文件

    As stated in the warnings, Transformers v4 support is to be completely removed in vLLM 0.24.0. Now is the right time to do this as v0.24.0 is the next release to be cut from main. This PR removes support for installing Transformers v4. Support for models created with Transformers v4 should be maintained. This also allows us to use a much more flexible Transformers pin in requirements/common.txt so…

  • 86111c0 #45191 — [Chore] Add Github notification for MRv2 for @yewentao256 (#45191)
    • 作者: Wentao Ye | +1/-1 | 1 个文件

    Add Github notification for MRv2 for @yewentao256

  • 82d6b59 #44687 — [CI/Build] Skip test_use_trtllm_attention on non-CUDA platforms (#44687)
    • 作者: Dan Blanaru | +7/-0 | 1 个文件

    Add a module-level skip so the test only runs on CUDA. module-level pytest.skip(…, allow_module_level=True) guarded on not current_platform.is_cuda(), so the file is skipped cleanly on non-CUDA platforms, matching the pattern already used by the sibling test_trtllm_kvfp8_dequant.py. This was requested by a @AndreasKaratzas on #43232 as an out-of-scope cleanup. bash pre-commit run –files tests/k…

  • 12f3f19 #35415 — feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions (#35415)
    • 作者: Nathan Price | +145/-13 | 3 个文件

    feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions ## Related work PR #35377 explored apply_chat_template for Qwen3-ASR; it was closed without merge. This PR keeps a narrower scope: wiring prompt through the API and sanitizing user text in the system turn. This PR also slots cleanly into the SpeechToTextParams refactor from #36268: get_generation_prompt reads request_prompt…

  • 6471ec7 #44978 — [EPLB] Reject NCCL-based EPLB communicators with async EPLB (#44978)
    • 作者: Ilya Markov | +37/-32 | 8 个文件

    NCCL is fundamentally incompatible with async EPLB due to multi-stream conflicts (see pytorch/pytorch#174288). Previously, the auto-selection logic avoided torch_nccl for async EPLB, but users could still explicitly set communicator=“torch_nccl” or communicator=“pynccl” and hit hangs. This PR adds explicit validation to catch these invalid combinations early The DeepEP low-latency + async EPLB cas…

  • ffce72c #44568 — [Model Runner V2] Fix v2 AttributeError: 'CohereASRDecoder' object has no attribute 'embed_input_ids' (#44568)
    • 作者: Wentao Ye | +4/-1 | 1 个文件

    Fixing for https://github.com/vllm-project/vllm/pull/44443 VLLM_USE_V2_MODEL_RUNNER=1 pytest tests/models/test_initialization.py::test_can_initialize_large_subset[CohereAsrForConditionalGeneration] Originally Now

  • dc66e01 #37898 — [Hybrid] Marconi-style admission policy for hybrid cache (#37898)
    • 作者: Stan Wozniak | +89/-0 | 3 个文件

    This PR implements Marconi-style admission policy for hybrid cache used e.g. for Qwen models. In Marconi paper an effective cache admission policy is proposed that caches in two cases: 1. Last state - important for chat workloads 1. Shared prefix (cache after the 2nd hit: observe prompts, if a shared prefix is detected, then cache it) - important for system prompts, instructions, few-shot …

  • 0bae1d3 #44586 — [MRV2][Spec Decode] DFlash (#44586)
    • 作者: Benjamin Chislett | +868/-47 | 14 个文件

    Implement DFlash for ModelRunnerV2, with full cudagraph support. Performance result: MRV1: MRV2: Running Qwen3 8B FP8 + DFlash on 1xGB200. ~1.2x speedup, from 5.2ms to 4.3ms per step (on independent benchmark, separate from profiling run). ## Testing Ran existing DFlash correctness testing with MRV1 and MRV2, got pass and identical acceptance rates. Enabled E2E DFlash correctness and AR regression…

  • 4673ca1 #44821 — fix: prefix DeepSeek V4 MTP projections (#44821)
    • 作者: Yufeng He | +4/-0 | 2 个文件

    Fixes #44817. - pass explicit prefix= values to the DeepSeek V4 MTP e_proj and h_proj ReplicatedLinear layers - apply the same fix to both NVIDIA and AMD DeepSeek V4 MTP implementations Without the prefixes, compressed-tensors receives an empty layer_name while constructing the draft model and cannot match artifact-side ignore/target rules. ## Duplicate work check I searched open, closed, and rece…

  • de900fa #45059 — fix: AOT compile cache collision for dataclass-based HF configs (#45059)
    • 作者: Angela Yi | +12/-12 | 1 个文件

    Fixes: pytorch/pytorch#184549 which we ran into when bumping vllm to torch 2.12 In transformers 5.x made PretrainedConfig a dataclass which led to normalize_value() serializing the HF configs via the dataclass branch instead of the attached to_json_string branch. This misses dynamically-set attributes like rope_parameters and rotary_kwargs that affect buffer shapes. #41277 then exposed this latent…

  • af65e08 #44193 — KV-Cache multi-tier offloading async batched lookup (#44193)
    • 作者: Effi Ofer | +518/-31 | 6 个文件

    Add async and batch lookup to multi-tier offloading. This significantly speeds up the secondary tier overall performance. - Run the kv_offloading/tiering tests concentrating on fs secondary tier as well as object store secondary tier (obj store PR has yet to be merged). - Run inference perf benchmark using llm-d-benchmark. ## Test Result # .venv/bin/python -m pytest tests/v1/kv_offload/test_fs_tie…

  • 3cc9fec #45131 — Deprecated 1st generation Qwen and QwenVL models (#45131)
    • 作者: Harry Mellor | +6/-1349 | 27 个文件

    These models: - have been superseded by several newer generations of Qwen architectures - are not used as building blocks for other architectures - incur maintenance burden They may still be used in versions of vLLM <= 0.23

  • 77f42d9 #45127 — [Model] Remove obsolete ERNIE models (#45127)
    • 作者: Tiezhen WANG | +4/-309 | 9 个文件

    Removes ErnieModel, ErnieForSequenceClassification, and ErnieForTokenClassification, the BERT-like Chinese ERNIE-3.0 pooling stack originally added in #36385. The implementation lived in a self-contained ernie.py with no external imports. Per vllm-release telemetry over the last 6 months: ErnieModel recorded 31 instances / 0.56 GPU-hours, vs. the Ernie 4.5 family’s combined ~76,000 GPU-hours over …

  • 9dfc313 #35669 — Feature/offloading manager stats (#35669)
    • 作者: Srinivas Krovvidi | +1200/-213 | 14 个文件

    This PR adds support for telemetry emissions from the OffloadingManager interface inside the KV Connector. As part of isolating the changes from the block-reuse frequency tracking PR, the OffloadingManager interface requires a standardized method to emit statistics up to the scheduler’s KVConnector interfaces. This change: 1. Adds [get_stats() -> dict[str, Any]](cci:1://file:///d:/Antigravity_Chec…

  • 8a5cf1c #44744 — [Security] Fix remote DoS via invalid recovered token reinjection (#44744)
    • 作者: Juan Pérez de Algaba | +99/-1 | 2 个文件

    Fixes https://github.com/vllm-project/vllm/security/advisories/GHSA-8wr5-jm2h-8r4f When vocab_size is not a multiple of BLOCK_SIZE (8192), the last Triton tile extends beyond the vocabulary. If all valid entries in that tail tile have zero target probability, the out-of-range masked positions (score 0) could win the tl.max tie-break, producing recovered_id >= vocab_size. This results in a remote d…

  • bd2d83f #39419 — [SpecDecode] Reduce TP communication for large-vocab draft models speculative decoding (#39419)
    • 作者: yiheng | +53/-39 | 7 个文件

    Use local argmax + gather reduction instead of all-gathering full-vocabulary logits to reduce TP communication overhead in speculative decoding modes such as DFlash and PARD. Enabling use_local_argmax_reduction improves performance in all tested scenarios. In terms of Total token throughput (tok/s), the gains are: * +9.38% for Qwen3-8B + Qwen3-8B-DFlash + tp2 at 1 concurren…

🐛 Bug Fix

  • 23eb7c8 #44422 — [Bugfix] Fix NixlEPAll2AllManager’s dependency on –enable-elastic-ep to function (#44422)
    • 作者: fangyuchu | +7/-1 | 1 个文件

    NixlEPAll2AllManager unconditionally asserts that tcp_store_group is not None in its init, but the standard EP path (without –enable-elastic-ep) never creates a tcp_store_group. This means nixl-ep cannot be used as an all2all backend unless elastic EP is also enabled, which is an unintended coupling. ## Root Cause The initialization chain: 1. CudaCommunicator.init receives tcp_store_g…

  • f1d8d99 #43495 — [Bugfix] CohereModel.load_weights: skip modelopt _quantizer.* keys (#43495)
    • 作者: Kai K. | +5/-1 | 1 个文件

    ModelOpt NVFP4 exports of Cohere / Command-A models include raw quantizer-module state keys named *.weight_quantizer._double_scale (and similar). These are modelopt calibration-time artifacts — superseded at runtime by the exported weight_scale / weight_scale_2 parameters — but they are not registered parameters on the vLLM-side CohereModel. CohereModel.load_weights then raises a KeyError when it …

  • 750aab5 #44424 — [Bugfix] Fix CPU memory leak related to not cleaning up old remotes data (#44424)
    • 作者: Nicolò Lucchesi | +192/-8 | 4 个文件

    Problem When a Prefill pod dies, the Decode pod never cleans up per-engine state accumulated during handshake (D has no way to know when a P dies/is not used anymore right now). These grow monotonically until D shuts down, leaking both NIXL resources and memory. ## Approach This PR adds a TTL-based eviction policy: if D hasn’t read from a remote engine for engine_ttl seconds (default 600, c…

  • 3508cb7 #43300 — [Bugfix] Fix broken profile_modular_kernel.py (#43300)
    • 作者: x41lakazam | +99/-11 | 2 个文件

    Make tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py usable again. ## Explanation The profile_modular_kernel utility has been unrunnable on main since the modular-kernel cleanup commit 8ad7285e ([Kernels] Clean up FusedMoeMethodBase and modular kernel setup, #22035). Every invocation fails with TypeError: FusedMoEKernel.apply() got an unexpected keyword argument ‘w1_scale’, regard…

  • 1f9dd79 #44680 — [Bugfix][Rust Frontend] Validate out-of-vocab token ids in request params (#44680)
    • 作者: Ting SUN | +258/-2 | 16 个文件

    With VLLM_USE_RUST_FRONTEND=1, allowed_token_ids, logit_bias keys, and /v1/completions token-id prompts are forwarded to the engine unchanged. An id at or beyond the model vocabulary makes the engine index the embedding table or the logits vector out of range, triggering a CUDA device-side assert that aborts EngineCore and takes down the whole server — a single well-formed request is a DoS. Th…

  • 1c3a72b #45180 — [Bugfix] Add fetch_images to MistralCommonImageProcessor (#45180)
    • 作者: Julien Denize | +81/-0 | 2 个文件

    Fix usage of mistral models going through MistralCommonImageProcessor that require fetch_images a recent method added by Transformers >= 5.10. Built upon https://github.com/vllm-project/vllm/pull/44989 Launched a ministral3 server and added unit test. ## Test Result all pass —

  • 248e33c #44608 — [Bugfix][Responses API] Set id on function_call item in streaming done event (#44608)
    • 作者: ankrovv | +1/-1 | 1 个文件

    Fixes the Harmony streaming function-call completion item to set id instead of passing event-only fields into ResponseFunctionToolCall. Previously, item_id was stored as an extra field and the real id stayed None, producing id: null in response.output_item.done. This matches the existing emit_simple_tool_call_done pattern.

  • b038a2f #45209 — [CI][Bugfix] Update Dockerfile dependency graph PNG (#45209)
    • 作者: Flora Feng | +0/-0 | 1 个文件

    Fix the Update Dockerfile dependency graph pre-commit hook that is currently failing on main. ## Bug The pre-commit hook update-dockerfile-graph regenerates docs/assets/contributing/dockerfile-stages-dependency.png whenever docker/Dockerfile changes. PR #44981 modified docker/Dockerfile but did not regenerate the PNG. This causes the hook to fail on every PR that runs pre-commit run –all-files –…

  • 2d481f8 #45025 — [Bugfix][Rust Frontend] Stop unescaping XML-style tool-call parameter values (#45025)
    • 作者: Ting SUN | +36/-120 | 6 个文件

    The minimax_m2, qwen_coder, glm_xml, and deepseek_dsml parsers run each parameter value through xml_unescape, but nothing in the pipeline ever escapes the value, so the decode only corrupts. A write_file argument of Tom & Jerry <3 comes back as Tom & Jerry <3. The Rust frontend’s own renderer emits these values raw, on purpose: the tojson filter deliberately does not HTML-escape (renderer/h…

  • 7920ccb #45067 — [Bugfix]: Fix Quark gpt-oss weight loading broken by FusedMoe refactor (#45067)
    • 作者: Rohan Potdar | +7/-0 | 1 个文件

    Broken by #41184; the router wieght and bias is now moved under the RoutedExperts class ## Test Result —

  • 5b6b536 #44679 — [ROCm][Bugfix] Make intermediate_pad TP-aware in rocm_aiter_fused_experts (#44679)
    • 作者: Rohan Potdar | +12/-2 | 1 个文件

    Fix the GSM8K accuracy regression on the AITER MoE path at TP>1 introduced by #42098. This is because FlyDSL and CKTile MoE follow different conventions for hidden_pad/intermediate_pad in AITER, and the aiter.fused_moe dispatcher uses FlyDSL on TP1 but CKTile on TP8. This will partially be fixed on the AITER side in 0.1.15 by https://github.com/ROCm/aiter/pull/3401. On openai/gpt-oss-120b BF16 at …

  • bfe1001 #45169 — [Bugfix] [DSV4] [ROCm] Pin apache-tvm-ffi version to 0.1.10 (#45169)
    • 作者: TJian | +3/-1 | 2 个文件

    It will cause the following error when launching deepseekv4 This is caused by the apache-tvm-ffi unintendedly updated to 0.1.12 . Pinning to the version 0.1.10 as what has been shown in the requirements/test/rocm.txt resolves this issue. Launch deepseekv4 successfully ## Test Result GSM8K Draft acceptance rate ### Docker image Log The image built is correctly pinned apache-tvm-ffi==0.1.10 —

  • fa8c868 #45047 — [Bugfix] Fix Llama4 weight loading (#45047)
    • 作者: Tyler Michael Smith | +18/-6 | 4 个文件

    The MoE refactor (#41184) moved expert weights under a routed_experts submodule but the remap function only handled .mlp.experts. paths. Models using .feed_forward.experts. (Llama4, mllama4, lfm2_moe) hit KeyError on scale params like w2_input_scale during weight loading. Generalize maybe_remap_moe_expert_param_name to match any .experts. pattern and add remap calls in the affected models’ weigh…

  • d1bcb4b #45147 — [Bugfix] Fix tool parsing crash with non-function tool types (e.g. WebSearchTool) (#45147)
    • 作者: Ben Browning | +48/-3 | 2 个文件

    Codex CLI’s default tool definitions include non-function types such as web_search. When the model generates a tool call in their presence, several popular tool parsers crash with a TypeError. Non-function tools are now filtered out of the helpers used for JSON schema generation and type coercion in function tool calls. ## Test Result 3 new unit tests all pass. AI assistance was used in the creati…

  • 2902668 #45037 — [Bugfix] Fix nemotron accuracy drop introduced by #41184 (#45037)
    • 作者: bnellnm | +3/-1 | 1 个文件

    The routed_scaling_factor was getting applied twice in the Nemotron model. We needed to check apply_routed_scaling_factor_to_output before passing the scaling factor onto the RoutedExperts. ## Test Result Test passed. —

  • 166d14e #45072 — [bugfix] skip conch kernel for g_idx reordering (#45072)
    • 作者: Divakar Verma | +6/-0 | 1 个文件

    Conch kernel doesn’t support g_idx reordering. The issue is not ROCm specific. CUDA fails to capture this because the kernel ordering on cuda picks Marlin kernel before conch which handles g_idx reordering. Kernel ordering preference for ROCm & CUDA can be referred here - LINK Resolves the following test: lora/test_quant_model.py::test_quant_model_lora[model0]. This PR will safely skip selec…

  • ccc05de #45073 — [Bugfix] Fix missing sequence_lengths in EXAONE-4.5 vision encoder (#45073)
    • 作者: Jongsu Liam Kim | +7/-0 | 1 个文件

    PR #42787 made the Qwen2.5-VL vision backbone pass sequence_lengths (FlashInfer CuDNN metadata) to every vision block, but the EXAONE-4.5 overrides of the vision block and attention kept the pre-#42787 signature. Since EXAONE-4.5 inherits Qwen2_5_VisionTransformer.forward, any multimodal request now fails with: TypeError: Exaone4_5_VisionBlock.forward() got an unexpected keyword argument ‘sequence…

  • c9e5bf8 #44814 — [Bugfix] Fix layerwise reload dropping params after a composed weight loader (#44814)
    • 作者: hallerite | +96/-1 | 2 个文件

    Fixes a bug in the layerwise weight-reload path (vllm/model_executor/model_loader/reload) that drops a trailing parameter of a layer when an earlier parameter uses composed_weight_loader. The reload path counts loaded elements per layer (CopyCounter inside get_numel_loaded) and finalizes a layer once load_numel >= load_numel_total. composed_weight_loader copies into its parameter twice — the i…

  • 87c15d4 #44921 — [Bugfix] Lazily import the humming quantization backend (#44921)
    • 作者: Michael Goin | +86/-52 | 5 个文件

    The quantization registry eagerly imports every backend, including humming. Importing humming has import-time side effects, so this happens for every quantized model even when humming isn’t used (see #44904). This adds a small lazy facade at vllm/utils/humming.py and routes all humming access through it, so the package is only imported when a humming layer is actually used. New humming symbols are…

  • 4882fd7 #39091 — [Bugfix][Reasoning] Nemotron V3: surface reasoning as content when thinking is unterminated (#39091)
    • 作者: Andrii Skliar | +163/-10 | 3 个文件

    Problem With Nemotron V3 and enable_thinking=False / force_nonempty_content=True, the model sometimes streams its whole answer inside an unterminated block (it never emits ). Clients that read only content (not reasoning) then get null. ## Fix vllm/reasoning/nemotron_v3_reasoning_parser.py - _should_force_content(request) — centralizes the opt-in check (request.chat_temp…

  • 9ad08c4 #44683 — [Bugfix][Rust Frontend] Fix missing added tokens in hf/fastokens tokenizer (#44683)
    • 作者: Isotr0py | +215/-8 | 4 个文件
    • For Qwen/Qwen2-VL-2B-Instruct, its image_token_id “<|image_pad|>” located at tokenizer_config.json but not tokenizer.json - So it will cause an error when create chat/text backends: - This PR fixes it by adding extra added tokens from tokenizer_config.json. ## Test Result —
  • a1ec011 #39498 — [Bugfix] Add deepseek_v32 to Quark dynamic MXFP4 model type check (#39498)
    • 作者: Shantipriya Parida | +32/-0 | 1 个文件

    Add deepseek_v32 to _DEEPSEEK_V3_FAMILY_MODEL_TYPES so that Quark correctly enables dynamic_mxfp4_quant for amd/DeepSeek-V3.2-mxfp4. Without this fix, excluded layers (e.g. self_attn) are not dynamically re-quantized when serving MXFP4-quantized DeepSeek-V3.2 checkpoints, causing silent correctness degradation (all-zero or garbage output tokens). ## Root cause _DEEPSEEK_V3_FAMILY_MODEL_TYPES is de…

  • fe1d923 #45110 — [BUGFIX][XPU] fix xpu flash_attn_varlen_func interface (#45110)
    • 作者: Kunshang Ji | +3/-0 | 1 个文件

    xpu path broke after https://github.com/vllm-project/vllm/pull/42175, which introduce a new paramter in flash_attn_varlen_func ## Test Result —

  • af9f583 #45029 — Revert “[Bugfix][CI] Gemma3 Transformers multimodal encoder profiling and build prompt-embedding fixtures” (#45029)
    • 作者: Harry Mellor | +1/-29 | 1 个文件

    Reverts the Transformers backend part of vllm-project/vllm#44952 It was not the correct fix for this problem. https://github.com/vllm-project/vllm/pull/39425 was.

📖 Documentation

  • c2b4cd3 #37047 — [Doc][Attention] Fix MLA top-of-file comments (#37047)
    • 作者: wineandchord | +10/-10 | 1 个文件

    Fix several obvious issues in the MLA top-of-file explanatory comment and pseudocode. ## Details - describe prefill as having a larger Sq / Skv ratio, close to 1 - describe decode as having a smaller Sq / Skv ratio, close to 0 - rename spda_o to sdpa_o in the pseudocode examples - fix the ql_nope pseudocode line to use q_nope - fix casual to causal in the chunked prefill pseudocode - remove the st…

  • 3d300ae #39400 — [Doc] Switch K8S examples to default MP mode (#39400)
    • 作者: Peter Pan | +472/-224 | 3 个文件

    PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED. Fix https://github.com/vllm-project/vllm/issues/38113 per @ed-pai, previous k8s yaml won’t work since default docker images has removed ray. so switch the k8s yaml example to default mp(multiprocessing) as distributed backend. ### Background: - since v0.19.0 (per https://github.com/vllm-projec…

🔧 Refactor

  • 3a04061 #45190 — [Refactor][Parser] Unify Response API to use parser.parse() like Chat Completion API (#45190)
    • 作者: Flora Feng | +101/-211 | 6 个文件

    Context The Chat Completion API (chat_completion/serving.py:952) uses parser.parse() for non-streaming output parsing, which returns (reasoning, content, tool_calls). The caller then constructs Chat-specific response objects from those results. The Response API (responses/serving.py:1037) uses a different method parser.extract_response_outputs(), which combines parsing AND ResponseOutputIte…

  • 32daf56 #45011 — [Refactor] Rename rocm_moe.py to rocm_moe_rdna.py (#45011)
    • 作者: JartX | +19/-19 | 3 个文件

    Renames the compressed-tensors MoE method module from rocm_moe.py to rocm_moe_rdna.py requested by @tjtanaa

⚡ Performance

  • e2db022 #45074 — [Perf][Attention] Pin MLA chunked-context metadata tensors so H2D copies are truly non-blocking (#45074)
    • 作者: qizixi | +5/-3 | 1 个文件

    Fix a hidden host-side stall in the MLA chunked-prefill metadata build that serializes CPU metadata prep with GPU execution on every scheduler step. In MLACommonMetadataBuilder.build(), the chunked-context section creates three small CPU tensors and ships them to the GPU with .to(device, non_blocking=True) when assembling ChunkedContextMetadata: - cu_seq_lens_cpu — allocated with pin_memory=True ✅…

  • 6850839 #44217 — [Perf] Fix dsv3_router_gemm heuristic (#44217)
    • 作者: Roberto L. Castro | +8/-6 | 1 个文件

    Observed in https://github.com/vllm-project/vllm/pull/42562 ## B300 microbenchmarks - DSv3 (K=7168, N=256): ## B300 microbenchmarks - DSv3 (K=7168, N=384): ## B200 microbenchmarks - DSv3 (K=7168, N=256): ## Benchmarking script

🧪 CI/Tests

  • 16282a9 #45170 — [ROCm][CI] Moving MI300 tests to MI325 until cluster is stabilized (#45170)
    • 作者: Andreas Karatzas | +17/-17 | 7 个文件

    Moving everything to MI325 due to MI300 outage.

  • 2ba68d9 #44946 — [Test] Fix one-sided MNNVL alltoall test workspace under-reservation (#44946)
    • 作者: Yongye Zhu | +5/-0 | 1 个文件

    tests/distributed/test_mnnvl_alltoall.py::test_one_sided_dispatch_combine initializes the FlashInfer one-sided MoeAlltoAll workspace without declaring the fp8 block-scale payload it later dispatches. The worker dispatches four payloads: - a1q — nvfp4 hidden states, (tokens, hidden // 2) → hidden // 2 B/token - a1q_scale — fp8 block scales, (tokens, hidden // 16) → hidden // 16 B/token - topk_i…

  • 2131b59 #45153 — [CI] Ping Mistral team for ministral/voxtral/mixtral/pixtral changes (#45153)
    • 作者: Julien Denize | +4/-4 | 1 个文件

    This PR change a bit the heuristic to ping the Mistral team on relevant file changes. Hopefully this will make us more responsive to issues and PRs made by the community. ## Test Result —

  • fdfb256 #44981 — [Rust Frontend] [CI] Unify Rust artifact builds with setuptools-rust (#44981)
    • 作者: Bugen Zhao | +117/-90 | 11 个文件

    In #40848 when we introduced integration with Rust frontend, we moved the rust build into a separate stage on CI so that it can be cached & parallelized more easily. However, this led to an issue that we actually have two build paths for artifacts: one is manual cargo invoke on CI, and the other is setuptools-rust through setup.py. That is manageable for the current vllm-rs binary, but it becomes …

  • 82a4223 #44823 — [ROCm][CI] Defer AITER sampler import and isolate server test PYTHONPATH (#44823)
    • 作者: Andreas Karatzas | +117/-17 | 3 个文件

    This PR addresses a failure in AMD OpenAI API server shard on mi300_1. The server runs with ROCm AITER enabled. vLLM selected the AITER sampler, but the request used per-request generators, which AITER sampling does not support and already falls back to the native PyTorch sampler for. Before this PR, vLLM imported aiter.ops.sampling during TopKTopPSampler construction, before checking whether the …

🦀 Rust Frontend

  • 6ec7dcd #44448 — [Frontend][Metrics] Add vllm:tool_call_parser_invocations_total Prometheus metric (#44448)
    • 作者: yzong-rh | +167/-20 | 3 个文件

    Add a metric for tool parser activity so operators can see how often the parser runs and whether an invocation produced a tool call. This makes it easier to spot tool-calling regressions during model rollouts or runtime changes. This PR adds the vllm:tool_call_parser_invocations_total counter and records it in DelegatingParser for both non-streaming and streaming tool parser calls, with labels for…