共 76 个 commit，涉及 439 个文件，+25660/-9952 行变动。

概要

统计项 数值
Commit 数 76
变更文件 439
新增行数 +25660
删除行数 -9952

Commit 列表

📦 Other

  • a30addc #44055 — [Docs][KV Connector][NIXL] document KV Transfer stat logging and Prometheus metrics (#44055)
    • 作者: Sai Sridhar Tarra | +48/-0 | 1 个文件

    Partially addresses #41230. Adds a Metrics Reference section to docs/features/nixl_connector_usage.md explaining every field in the periodic KV Transfer metrics log line and all seven Prometheus metrics registered by NixlPromMetrics. The descriptions are derived directly from vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py (NixlKVConnectorStats.reduce() and NixlPromMetrics). ## What…

  • 9ff278b #43877 — [Core][KV Connector] fix scheduler KV connector stats aggregation (#43877)
    • 作者: Srinivas Krovvidi | +102/-8 | 3 个文件

    Extracts the scheduler KV connector stats ordering/aggregation fix from #35669 into a dedicated PR. This fixes scheduler-side KV connector stats collection so that: - update_connector_output() runs before scheduler-side stats are collected. - worker-side and scheduler-side KV connector stats are aggregated when both are present. - scheduler-only KV connector stats are still emitted when no worker-…

  • c7aa3d2 #35022 — [Core] Support structured outputs for beam search (#35022)
    • 作者: Guan-Ming (Wesley) Chiu | +405/-70 | 3 个文件

    Support structured outputs (e.g., JSON schema) for beam search. Previously, beam search had no way to constrain generation to valid JSON or other grammar-defined formats. This adds grammar-based constrained decoding to the beam search loop. Related: https://github.com/vllm-project/vllm/issues/34782 pytest tests/samplers/test_beam_search.py::test_beam_search_structured_output -v ## Test Result Loca…

  • fbc3a19 #42759 — [Bug] Migrate Reset cache for both v2 and v1 model runner (#42759)
    • 作者: Wentao Ye | +3/-2 | 2 个文件

    Migrate Reset cache for both v2 and v1 model runner Solve https://github.com/vllm-project/vllm/pull/42673#issuecomment-4461618459

  • 4171ae4 #39457 — [V1][Metrics] Add MLA attention metrics for DeepSeek MFU estimation (#39457)
    • 作者: Thillai Chithambaram | +600/-0 | 2 个文件

    Adds MLAAttentionMetrics to the V1 performance metrics module to correctly estimate FLOPs and memory bandwidth for models using Multi-Latent Attention (MLA), such as DeepSeek-V2, DeepSeek-V3, and DeepSeek-R1. The existing AttentionMetrics assumes standard MHA or GQA attention patterns with separate K/V projections and large KV cache representations. MLA introduces a fundamentally different archite…

  • b7f9b6a #42206 — [Metrics] Add group-aware KV cache capacity to vllm:cache_config_info (#42206)
    • 作者: Ethan Feng | +75/-26 | 7 个文件

    Addresses the Prometheus vs. startup-log discrepancy for KV cache capacity in #42024. The startup log already reports the correct group-aware KV cache capacity for hybrid models, but Prometheus did not expose matching info in ‘vllm:cache_config_info. This PR adds two: - kv_cache_size_tokens : Per-DP-engine KV cache capacity in tokens (group-aware). Uses group-aware capacity since num_gpu_blocks * …

  • f1e13f7 #45129 — [Model] Remove Mono-InternVL (InternLM2VEForCausalLM) (#45129)
    • 作者: Tiezhen WANG | +53/-262 | 9 个文件

    Removes the InternLM2VEForCausalLM Visual-Expert variant used only by OpenGVLab/Mono-InternVL-2B (~415 HF downloads, model abandoned upstream). Per vllm-release telemetry: 0 instances / 0 GPU-hours across the last 6 months of releases. The model is also blocked under Transformers v5 (the test registry pinned it to max_transformers_version=“4.57”). Also strips the Mono-InternVL branches from Intern…

  • 88ed636 #35264 — [KV Connector]: Support KV push from Prefill to Decode node using Nixl KV Connector (#35264)
    • 作者: snadampal | +6335/-3370 | 26 个文件

    RFC https://github.com/vllm-project/vllm/issues/36923 Implemented KV push feature where Prefill node pushes its KV blocks to Decode node as soon as the model executor completes the forward pass and finishes request. The implementation supports heterogeneous TP and heterogeneous block sizes between P and D nodes And it covers both the scenarios: Scenario 1: D registers blocks with P before P finish…

  • a014ddd #45304 — [11b/n] Migrate Machete kernels to torch stable ABI (#45304)
    • 作者: Chris Leonard | +341/-323 | 17 个文件

    This PR continues the libtorch stable ABI migration (see #26946) for vLLM Machete CUDA kernels by moving the Machete tree (host entry point, launchers, CUTLASS/CuTe device headers, and codegen) into csrc/libtorch_stable/quantization/machete/, registering machete_mm, machete_prepack_B, and machete_supported_schedules on _C_stable_libtorch, and removing them from legacy _C. CMake Machete generation …

  • f715f25 #45113 — Fix misleading error for audio duration limit rejection (#45113)
    • 作者: Juan Pérez de Algaba | +11/-6 | 2 个文件

    Let the descriptive ValueError from load_audio propagate directly instead of catching it and re-raising a generic “Invalid or unsupported audio file” message. Also update the error messages to mention the VLLM_MAX_AUDIO_DECODE_DURATION_S env var so users know how to increase the limit.

  • 462ef83 #45294 — Update hidden states extraction integration test triggers (#45294)
    • 作者: Fynn Schmitt-Ulms | +15/-0 | 1 个文件

    The example_hidden_states_connector.py file is used significantly by the hidden states extraction system. This pr ensures that changes to it also trigger the integration tests. This pr also adds a second job that runs just the multi-gpu tests in tests/v1/kv_connector/extract_hidden_states_integration which are currently not run. #43805 added a test that requires TP=2 but didn’t add this trigger. E…

  • 04cec9e #45240 — [XPU][DeepSeek-V4] Fix MTP: sync with upstream fixes #44821 and #43746 (#45240)
    • 作者: Ma Jian | +29/-18 | 1 个文件

    Sync the XPU DeepSeek V4 MTP implementation with two upstream fixes: 1. #44821 — fix: prefix DeepSeek V4 MTP projections - Pass explicit prefix= to e_proj and h_proj ReplicatedLinear layers so compressed-tensors can match artifact-side ignore/target rules. 2. #43746 — [Model Refactoring] Remove torch compile dependency in DSv4 - Remove @support_torch_compile decorator, align with breakable…

  • 0cd9b7a #43409 — [CPU] Support CPU W4A16 INT4 MoE (#43409)
    • 作者: Yuwen Zhou | +685/-22 | 8 个文件

    Support cpu AWQ, GPTQ & compressed-tensor W4A16 INT4 MoE for Intel CPUs. python -m pytest tests/kernels/moe/test_cpu_quant_fused_moe.py -v ## Test Result 1. AWQ W4A16 INT4 MoE python examples/basic/offline_inference/generate.py –model QuixiAI/Qwen3-30B-A3B-AWQ –max-model-len 512 –enforce-eager 2. GPTQ W4A16 INT4 MoE python examples/basic/offline_inference/generate.py –model Qwen/Qwen3-…

  • fe04238 #44893 — [ROCm][gpt-oss] Pass GateMode.INTERLEAVE for MXFP4 W4A16 fused MoE (#44893)
    • 作者: Rohan Potdar | +40/-0 | 2 个文件

    Fixes gpt-oss-120b accuracy=0 on AITER ≥0.1.14 and the gpt-oss-20b-w-mxfp4-a-bf16 CK2stages JIT crash: https://github.com/ROCm/aiter/issues/3586 oracle/mxfp4.py shuffles MXFP4 W4A16 weights with shuffle_weight_a16w4 (interleaved gate/up), but the call into aiter.fused_moe was using the default GateMode.SEPARATED. Before ROCm/aiter#3123 the aiter dispatcher tolerated this; #3123 added gate-mode-sen…

  • eb28452 #45163 — [Model] Add DiffusionGemma Support (#45163)
    • 作者: Lucas Wilkinson | +2698/-235 | 52 个文件

    See: https://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-it Docker images are available under: vllm-openai:gemma-cu130

  • 6fbfdd1 #44583 — [NIXL] Per-region KV transfer classification for mixed full-attn + MLA groups (#44583)
    • 作者: Dao007forever | +234/-68 | 3 个文件

    Within a single KV-cache group the NIXL connector may need to transfer regions of two kinds: - Full-attention (GQA) layers, whose KV is head-sharded across TP → SPLIT (each rank reads its head slice from the remote at a per-rank offset). - MLA layers, whose latent KV is replicated on every rank and is key-only → REPLICATE (whole block read from one rank at offset 0, no V stream). P…

  • 7021be6 #45176 — [11a/n] Migrate Marlin kernels to torch stable ABI (#45176)
    • 作者: Chris Leonard | +627/-573 | 19 个文件

    his PR continues the libtorch stable ABI migration (see #26946) for vLLM Marlin CUDA kernels by moving Marlin host ops (marlin_gemm, gptq_marlin_repack, awq_marlin_repack, and marlin_int4_fp8_preprocess), generated template kernels (sm75/sm80/sm89), and shared device headers into csrc/libtorch_stable/quantization/marlin/, registering them on _C_stable_libtorch and removing them from legacy _C. Thi…

  • 226ba9f #44587 — [ASR] Add Long Audio benchmark and correctness test (#44587)
    • 作者: Ekagra Ranjan | +530/-46 | 5 个文件

    Our ASR testing (correctness and RTFx benchmarking) has 2 shortcomings: 1. Correctness: we test only on short audio duration samples for ASR correctness - doesnt gate the code path for long audio RMS chunking 2. RTFx bench - the supported ASRDatasets are 18-100GB in size (even though final split is test) which makes measuring RTFx expensive. Plus it has a lot of short duration samples This PR adds…

  • e0b9fb1 #44612 — [ASR] Optimize CPU preproc to get 2.5x RTFx via multi-threading (#44612)
    • 作者: Ekagra Ranjan | +105/-34 | 4 个文件

    Currently, vLLM ASR’s CPU preprocessing like audio loading and RMS chunking is a synchronous code which blocks the main eventloop of FastAPI server which leads to serial execution instead of batched processing. This bottleneck becomes much more painful when the system is processing long audio . Under high concurrency, it also leads to timeouts in health/ endpoint which makes it difficult to identi…

  • 9bbf42b #45305 — Make mistral_common optional by deferring MistralToolCall import (#45305)
    • 作者: Neil Schemenauer | +3/-1 | 1 个文件

    Defer the import of MistralToolCall so that we don’t import mistral_common unless we actually need it. This makes mistral_common an optional dependency and makes startup faster in the case you don’t use it. The change defers the MistralToolCall import into the is_mistral_tokenizer branch, making mistral_common an optional dependency. Verified in an environment WITHOUT mistral_common installed: 1. …

  • 5a6c7b7 #45052 — [Bug] Fix test flashmla for DSv4 (#45052)
    • 作者: Wentao Ye | +5/-3 | 1 个文件

    pytest tests/kernels/attention/test_flashmla_sparse.py Will raise error This is due to upstream API interface change, this PR fixes the issue Now

  • c9340e6 #45128 — [Model] Remove InternLMForCausalLM registry alias (#45128)
    • 作者: Tiezhen WANG | +1/-17 | 14 个文件

    The InternLMForCausalLM architecture string was a one-line alias mapping to LlamaForCausalLM. The original InternLM-v1 line is superseded by InternLM2/3, and the alias is no longer maintained. Per vllm-release telemetry over the last 6 months: 44 instances / 2.31 GPU-hours, vs. InternLM2ForCausalLM’s 30,904 GPU-hours over the same window (~0.007%). Users with config.json declaring “InternLMForCaus…

  • 3b03a2c #43965 — [Rust Frontend] Support continuous_usage_stats stream option (#43965)
    • 作者: Chao-Ju Chen | +335/-51 | 9 个文件

    This PR adds Rust frontend support for the OpenAI-compatible stream_options.continuous_usage_stats request option for both chat completions and completions. The Rust routes already supported stream_options.include_usage, but still rejectedcontinuous_usage_stats. This change wires the request option through request preparation and streaming response generation, matching Python’s should_include_usag…

  • 79f8c5b #42331 — [Metrics] Scope unregister_vllm_metrics() to strictly “vllm:” metrics (#42331)
    • 作者: vraiti | +1/-1 | 1 个文件

    unregister_vllm_metrics() currently uses “vllm” in collector._name to decide which collectors to remove from the Prometheus registry. This is a substring match that removes every collector whose internal name contains vllm anywhere, including metrics registered by other subsystems or downstream extensions that happen to use vllm in their metric names. This came up when trying to add Prometheus s…

  • f81daf8 #41797 — [Attention] add triton diff-kv backend for mimo (#41797)
    • 作者: Jiangyun Zhu | +1041/-9 | 8 个文件

    Fix https://github.com/vllm-project/vllm/issues/41519 ## Test Result FA |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|——:|—————-|—–:|———–|—|—–:|—|—–:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9378|± |0.0067| | | |strict-match | 5|exact_match|↑ |0.9371|± |0.0067| triton |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|—-…

  • 4085ff7 #44594 — [Core] Add kvcache watermark to reduce preemptions (#44594)
    • 作者: Nick Hill | +291/-8 | 7 个文件

    Configurable percentage of total kvcache blocks to reserve as buffer when allocating new request blocks. Disabled by default. Does not apply to first new request when cache is empty. #### Benchmark Config - Model / HW: Qwen/Qwen2.5-7B-Instruct, TP=1, 1× NVIDIA GB200 - KV cache: 16 GiB → 299,584 tokens (near-critical: ~1.5× mean demand), prefix caching off - Engine: –max-model-len 8192…

  • 5edf7ff #45179 — [Core] Release cached device memory under pressure on UMA GPUs during weight loading (#45179)
    • 作者: Michael Goin | +42/-0 | 2 个文件

    Alternative to #44502. Weight repacking in process_weights_after_loading leaves freed checkpoint-format tensors in the caching allocator; on integrated (UMA) devices like DGX Spark the reserved cache starves the OS, which can thrash before an OOM retry would release it. Differences from #44502: - One call site in the loader loop covers all quant methods, not just NVFP4 Marlin - No per-layer gc.col…

  • b78fc47 #45218 — [Docs] Add redirect for moved lmcache examples page (#45218)
    • 作者: Natalie Lin | +1/-0 | 1 个文件

    Add a missing redirect for the examples/others/lmcache path, which was moved to examples/disaggregated/lmcache in #41082 but without a corresponding redirect entry in mkdocs.yaml. This causes a broken link for users arriving from search engines: the top Google result for “vllm lmcache” is the v0.10.1 docs page, whose “latest stable version” banner links to stable/examples/others/lmcache.html, whic…

  • 03878d1 #44992 — Deprecations for v0.23 and v0.24 (#44992)
    • 作者: Harry Mellor | +102/-676 | 35 个文件

    Perform deletions for deprecations scheduled for: - v0.23 - these should technically have already been deleted as v0.23 has already been cut - v0.24* - this will be the next minor release to be cut *this PR does not include the deletion of the Transformers v4 code path. This is somewhat more complicated and will be done in a follow up PR.

  • 55911db #44243 — [PD][Core] Fix Mamba prefix cache hit rate in PD disaggregation (#44243)
    • 作者: zhanqiuhu | +534/-3 | 5 个文件

    Co-authored with @underfituu. Fix the bug described in https://github.com/vllm-project/vllm/pull/42524; overwrite find_longest_cache_hit to bypass truncation of full attention groups. Note on the expected behavior: the last state of Mamba will always be transfer, but full attention will only transfer the prefix cache miss part.

  • cc640ee #45030 — [Rust Frontend][Metrics] Export vllm:lora_requests_info from frontend (#45030)
    • 作者: Will Eaton | +383/-36 | 8 个文件

    Add support for reporting LoRA adapter metrics to the rust frontend, this is required for llm-d compatibility with the lora-aware-routing affinity scorer. ref: https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/scheduling/scorer/loraaffinity The engine’s SchedulerStats never carries adapter names: in the python frontend they are computed frontend-side by LoRARequestStates in…

  • ebc6ef9 #43805 — Hidden states extraction improvements (#43805)
    • 作者: Fynn Schmitt-Ulms | +457/-236 | 7 个文件

    The current ExampleHiddenStatesConnector saves the hidden states in wait_for_save method of the KVConnector. The challenge with this, is that when running with chunked prefill, these method is called multiple times per request. The current solution is to just disable chunked prefill and enforce that when using the connector. This pr removes this requirement, by moving the save step into get_finish…

  • ab3a1fd #45244 — minicpmv4_6: fix ImageSize (W,H) order for placeholder token calculation (#45244)
    • 作者: tc-mb | +8/-81 | 2 个文件

    Fix a bug where _compute_visual_tokens passes ImageSize(width, height) directly to transformers image processor methods (get_sliced_grid, find_best_resize, get_refine_size) that expect (height, width) order. For certain aspect ratios (e.g. 365x211) this causes the text-side placeholder count to mismatch the actual vision embedding count, triggering a ValueError crash. - minicpmv4_6.py: convert Ima…

  • c3662b3 #44733 — [KV offload] Parallel-agnostic fs-tier cache for single full-attention group (#44733)
    • 作者: Itay Etelis | +102/-3 | 4 个文件

    Tests pytest tests/v1/kv_offload/tiering/test_fs_tier.py — adds 3 predicate cases (single full-attn → agnostic; multi-group → off; non-full-attn → off). Validated end-to-end on 4×H100 (store TP=2 → load TP=4, same cache dir): | Model | Result | |—|—| | Qwen2.5-7B (full attention) | 400/403 tokens loaded from the TP=2 cache; output identical to fresh TP=4 | | DeepSeek-V2-Lite (MLA) | Short r…

  • e62d00a #45253 — docs: add fix disclosure policy to SECURITY.md (#45253)
    • 作者: Juan Pérez de Algaba | +9/-0 | 1 个文件

    Document that MODERATE and LOW severity fixes are developed as public pull requests, while CRITICAL and HIGH severity issues use a private security fork. The vulnerability management team retains discretion to adjust the approach on a case-by-case basis.

  • 1f60771 #42679 — fix: guard flash-attn rotary import (#42679)
    • 作者: Yufeng He | +7/-5 | 1 个文件

    Fixes #42675. - import flash_attn.ops.triton.rotary directly when vLLM is not on CPU - treat a missing rotary module as unavailable instead of crashing during ApplyRotaryEmb construction - keep the existing fallback path when FA4 no longer provides the old rotary module FA4 can leave the flash_attn root package importable while moving or removing flash_attn.ops.triton.rotary. Checking only the roo…

  • 05d9848 #44923 — [Build] Upgrade CUDA Dockerfiles from GCC 10 to GCC 12 for C++20 compatibility (#44923)
    • 作者: Richard Barnes | +27/-9 | 4 个文件

    Why this is not a duplicate Checked open PRs on 2026-06-08 with gh pr list –search “gcc-10 gcc-12” and –search “c++20 compiler” — no existing PR addresses this. PyTorch PR #167929 raised the minimum C++ standard to C++20 and the minimum GCC version to 11.3. vLLM’s main CUDA Dockerfile and nightly-torch Dockerfile both pin GCC to version 10 to suppress spurious -Wredundant-move warning spa…

  • ef67071 #44783 — [Build] Skip spinloop extension on Python < 3.11 (#44783)
    • 作者: jasen | +17/-13 | 2 个文件

    The spinloop extension uses Py_buffer, PyBuffer_Release, and the “y*” format code in PyArg_ParseTupleAndKeywords — none of which are available in the Python Limited/Stable API. Combined with USE_SABI 3.11, this causes compilation to fail on Python 3.10 because the Limited API headers do not declare these symbols. Guard the spinloop target in both CMakeLists.txt (Python_VERSION check) and setup.py …

  • 432905d #45262 — Only enable PR docs builds manually (#45262)
    • 作者: Harry Mellor | +20/-6 | 1 个文件

    Even with the gating we added recently, queue times are still very long. This PR changes the gate to be fully manual based on the presence of a build-docs label on the PR.

  • 9492362 #45119 — [Security] Apply sanitize_message to Anthropic and STT error paths (#45119)
    • 作者: Juan Pérez de Algaba | +91/-5 | 4 个文件
    • Apply sanitize_message() to all 5 error response sites in the Anthropic API router, its SSE streaming converter, and the speech-to-text WebSocket paths that were missed by the original CVE-2026-22778 fix (PRs #31987, #32319). - These paths catch exceptions in-route and return str(e) directly, bypassing the global FastAPI exception handler that already applies sanitization. PIL UnidentifiedImageE…
  • 7852e50 #43724 — [docs] Document –scheduler-cls base class requirement (extend AsyncScheduler, not Scheduler) (#43724)
    • 作者: Georgii Kliukovkin | +6/-5 | 1 个文件

    Documents the requirement that custom scheduler plugins loaded via –scheduler-cls should extend AsyncScheduler, not Scheduler. The default V1 scheduler is AsyncScheduler; subclassing Scheduler instead disables async scheduling overlap with GPU execution and causes significant latency regression (we measured ~78% on multi-turn workloads before catching the mismatch). ## Context This footgun was id…

  • 0d657e4 #45155 — [Rust Frontend] Fix DeepSeek V3.2 continue_final_message rendering (#45155)
    • 作者: Reid | +22/-3 | 2 个文件

    Fix DeepSeek V3.2 Rust chat rendering for continue_final_message=true. Before this change, the DeepSeek V3.2 renderer ignored the final-assistant continuation mode and always passed prefix=false when rendering assistant messages. As a result, a final assistant prefill could be rendered as a completed assistant message with <｜end▁of▁sentence｜>, instead of remaining open-ended for the model to conti…

  • aa1df36 #44980 — Fix/minicpmv46 missing version (#44980)
    • 作者: 王金旭 | +32/-2 | 3 个文件

    Reported in OpenBMB/MiniCPM-V#1113 (vllm serve openbmb/MiniCPM-V-4_6). ## Fix Two layers: 1. MiniCPMV4_6ProcessingInfo.get_hf_processor() now returns the native MiniCPMV4_6Processor directly instead of the vendored wrapper. MiniCPM-V 4.6 already implements its own image/video handling and prompt-update logic, so it neither needs nor is compatible with the vendored processor designed for 2.x/4….

  • f06aefb #44523 — [CPU] Add missing scalar fallback for CPU W4A8 INT4 GEMM (#44523)
    • 作者: wcy | +61/-15 | 4 个文件

    Summary This PR fills in the scalar fallback implementation for the CPU W4A8 INT4 GEMM path. The scalar path is mainly intended as a correctness/reference fallback when optimized paths such as AVX512/brgemm are unavailable. As part of this, the INT4 W4A8 CPU ops are registered on RISC-V so the direct C++/Torch op path can be exercised there. Before this change, non-AVX targets that reached the…

  • d598d23 #45116 — [Security] Reject non-finite temperature and repetition_penalty values (#45116)
    • 作者: Juan Pérez de Algaba | +63/-0 | 2 个文件
    • Add math.isfinite() validation for temperature and repetition_penalty in SamplingParams._verify_args(). - NaN and Infinity bypass Python’s comparison operators (<, >) due to IEEE 754 float semantics, allowing them to propagate to GPU sampling kernels where they cause undefined behavior or CUDA crashes. - Addresses advisory GHSA-7h4p-rffg-7823.
  • f219788 #44971 — [Security] Fix info disclosure via int32 truncation in GGUF dequantize kernels (#44971)
    • 作者: Juan Pérez de Algaba | +33/-29 | 3 个文件

    Fix an information disclosure vulnerability caused by integer truncation in GGUF dequantize kernels (csrc/libtorch_stable/quantization/gguf/). The to_cuda_ggml_t function pointer typedef declares its element count parameter k as int (32-bit). When a GGUF model has weight tensor dimensions whose product exceeds INT_MAX (e.g. a 65536x65536 matrix), the int64_t product m * n is silently truncated to …

🦀 Rust Frontend

  • 3b8fc3f #45396 — [Frontend] Support strict mode for tool calling with ResponsesAPI (#45396)
    • 作者: Chauncey | +98/-23 | 5 个文件

    Follow up https://github.com/vllm-project/vllm/pull/45003 [Frontend] Support strict mode for tool calling with ResponsesAPI see e2e ## Test Result —

  • 2043258 #45003 — [Frontend] Support strict mode for tool calling (#45003)
    • 作者: Chauncey | +672/-1936 | 29 个文件

    Co-authored-by: @cjackal 44624812+cjackal@users.noreply.github.com [Frontend] Support strict mode for tool calling I tested it locally with Minimax 2.5,Qwen2.5, Qwen3.5, Qwen3.6, Qwen3, and DeepSeek V3.2. I also tested the tool_choice modes required, auto, and named tool selection, and all of them worked correctly. ## Test Result main: this pr —

🐛 Bug Fix

  • 8af550b #45394 — [BUGFIX][XPU] Update fa interface for compatibility (#45394)
    • 作者: liuzhenwei | +2/-0 | 1 个文件

    Test Result —

  • 1ae1051 #45286 — [Bugfix][Rust Frontend] Return 400 for prompt-validation submit errors (#45286)
    • 作者: JinYan Su | +89/-18 | 4 个文件

    Tokenized-prompt validation failures raised during request lowering (PromptTooLong, EmptyPromptTokenIds) are wrapped in the blanket server_error! at the three submit sites, so a client sending a prompt over max_model_len receives HTTP 500 server_error instead of the 400 invalid_request_error the Python frontend returns for the same mistake: Clients can’t distinguish “my request is invalid” from “t…

  • 87b98d6 #45300 — [Rust Frontend][Bugfix] Forward –shutdown-timeout and –disable-log-stats to the managed Python engine (#45300)
    • 作者: Will Eaton | +47/-0 | 3 个文件

    In vllm-rs serve (managed mode), recognized frontend flags are kept on the Rust side by the CLI splitter, so flags the spawned Python engine must also see have to be re-forwarded explicitly in into_config. Two flags were missing from that list: - –shutdown-timeout: the engine core always ran with its default shutdown_timeout=0 (abort mode), so on SIGTERM it killed all in-flight requests immediate…

  • a2c72d4 #45374 — [Bugfix] Fix Dockerfile dependency graph pre-commit error (#45374)
    • 作者: Isotr0py | +0/-0 | 1 个文件
  • b927004 #44599 — [Bugfix] Mamba CPU Offloading (#44599)
    • 作者: Varun Sundar Rabindranath | +114/-1 | 2 个文件

    CPUOffloading + Mamba models + cache-mode “align” : Compared to cold outputs (don’t hit the CPU), this combination yields incorrect results for cache’d prompts that end exactly at the offloaded block boundaries. When at block boundaries, the Offloading code path treats MambaSpec like sliding-window attention and returns hit-tokens that are not multiples of mamba cache size. This is incorrect. The …

  • 42ae5e7 #44383 — [Bugfix] Fix –enable-prompt-tokens-details omitting zero cached tokens (#44383)
    • 作者: sasindharan | +61/-9 | 5 个文件

    Fixes #44377 ## Problem When –enable-prompt-tokens-details is enabled, the OpenAI-compatible API omits prompt_tokens_details when num_cached_tokens == 0 because the conditions use truthiness checks (and num_cached_tokens) which treat 0 as falsy — the same as None. This makes it impossible for clients to distinguish between: - prompt token details enabled and cached_tokens: 0 - prompt token detail…

  • 2263f8a #45345 — [CI][BugFix] Fix broken test_mamba_prefix_cache.py due to stale mock (#45345)
    • 作者: Nick Hill | +2/-0 | 1 个文件

    Recent PR https://github.com/vllm-project/vllm/pull/44594 change the method signature of KVCacheManager.allocate_slots() which needs to be updated in a test mock. I don’t think the test in question wasn’t chosen to run in the PR’s CI which is why it was missed.

  • c107683 #45308 — [Bugfix][Model] Pass revision by name in Run:ai and bitsandbytes index downloads (#45308)
    • 作者: Ting SUN | +59/-3 | 4 个文件

    RunaiModelStreamerLoader._prepare_weights and BitsAndBytesModelLoader._prepare_weights call download_safetensors_index_file_from_hf with revision as the fourth positional argument. The callee signature is: So the requested revision is bound to subfolder, while the real revision parameter remains None. When a non default revision is requested, the safetensors index can be downloaded from the wrong …

  • 4bc8332 #44592 — [Bugfix] OffloadingConnector: respect skip_reading_prefix_cache flag (#44592)
    • 作者: littlecircle0730 | +61/-2 | 3 个文件

    Fix #44585 OffloadingConnectorScheduler.get_num_new_matched_tokens() was ignoring request.skip_reading_prefix_cache and querying the CPU offload cache unconditionally. This caused requests that explicitly opted out of prefix cache reads (e.g. those with prompt_logprobs set, or skip_reading_prefix_cache=True) to silently load KV blocks from CPU anyway, returning incorrect prompt_logprobs results an…

  • 6f573f4 #45217 — [Bugfix] Initialize missing attributes in mistral eagle (#45217)
    • 作者: jpwang | +156/-0 | 2 个文件

    Fixes #44846. EagleMistralLarge3Model subclasses DeepseekV2Model, but its init calls nn.Module.init(self) directly instead of DeepseekV2Model.init. It then calls super().forward(…) and inherits DeepseekV2Model.load_weights, so it must initialize the DeepseekV2Model attributes used by those paths. This PR initializes: - self.use_mha - self.num_redundant_experts using the same logic as…

  • 8a91228 #45206 — [Bugfix][KVConnector][Mooncake] Close MooncakeDistributedStore on connector teardown (#45206)
    • 作者: Dao007forever | +125/-0 | 4 个文件

    MooncakeStoreConnector never overrode the base KVConnectorBase_V1.shutdown() hook and had no del, so the worker’s MooncakeDistributedStore handle was never closed officially (we depends on aexit on C++ side to clean up). The store owns a TransferEngine, the registered RDMA buffers, and the connection to the master server — all of which leaked on every connector teardown (e.g. engine shutdo…

  • 235b63c #45287 — [Bugfix] Fix Anthropic tool_use content handling dropping args (#45287)
    • 作者: Ben Browning | +252/-5 | 2 个文件

    Content deltas (e.g. “\n” after </tool_call>) arriving in the same SSE chunk as incremental tool_call argument deltas caused stop_active_block() to reset tool_use_id before arguments were processed, silently dropping them. Buffer content during active tool_use blocks and flush it as a text block on the next block transition. This fixes an issue with multiple models for clients using our Anthropic …

  • b814229 #45251 — [Bugfix] Restrict FlashInfer cuDNN FP8 ViT attention gate to Blackwell (SM 100) (#45251)
    • 作者: wentian-byte | +14/-6 | 2 个文件

    is_flashinfer_cudnn_fp8_prefill_attn_supported() currently returns True on SM 90 GPUs (Hopper, e.g. H100 / H200 / H20), but the underlying cuDNN FP8 SDPA forward path requires Blackwell (SM 100) when the output dtype is bf16/fp16 — which is exactly the configuration MMEncoderAttention._forward_flashinfer uses (o_data_type=self.dtype, where self.dtype is the model dtype, typically bf16). As a resul…

  • 23eb7c8 #44422 — [Bugfix] Fix NixlEPAll2AllManager’s dependency on –enable-elastic-ep to function (#44422)
    • 作者: fangyuchu | +7/-1 | 1 个文件

    NixlEPAll2AllManager unconditionally asserts that tcp_store_group is not None in its init, but the standard EP path (without –enable-elastic-ep) never creates a tcp_store_group. This means nixl-ep cannot be used as an all2all backend unless elastic EP is also enabled, which is an unintended coupling. ## Root Cause The initialization chain: 1. CudaCommunicator.init receives tcp_store_g…

  • f1d8d99 #43495 — [Bugfix] CohereModel.load_weights: skip modelopt _quantizer.* keys (#43495)
    • 作者: Kai K. | +5/-1 | 1 个文件

    ModelOpt NVFP4 exports of Cohere / Command-A models include raw quantizer-module state keys named *.weight_quantizer._double_scale (and similar). These are modelopt calibration-time artifacts — superseded at runtime by the exported weight_scale / weight_scale_2 parameters — but they are not registered parameters on the vLLM-side CohereModel. CohereModel.load_weights then raises a KeyError when it …

  • 750aab5 #44424 — [Bugfix] Fix CPU memory leak related to not cleaning up old remotes data (#44424)
    • 作者: Nicolò Lucchesi | +192/-8 | 4 个文件

    Problem When a Prefill pod dies, the Decode pod never cleans up per-engine state accumulated during handshake (D has no way to know when a P dies/is not used anymore right now). These grow monotonically until D shuts down, leaking both NIXL resources and memory. ## Approach This PR adds a TTL-based eviction policy: if D hasn’t read from a remote engine for engine_ttl seconds (default 600, c…

  • 3508cb7 #43300 — [Bugfix] Fix broken profile_modular_kernel.py (#43300)
    • 作者: x41lakazam | +99/-11 | 2 个文件

    Make tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py usable again. ## Explanation The profile_modular_kernel utility has been unrunnable on main since the modular-kernel cleanup commit 8ad7285e ([Kernels] Clean up FusedMoeMethodBase and modular kernel setup, #22035). Every invocation fails with TypeError: FusedMoEKernel.apply() got an unexpected keyword argument ‘w1_scale’, regard…

  • 1f9dd79 #44680 — [Bugfix][Rust Frontend] Validate out-of-vocab token ids in request params (#44680)
    • 作者: Ting SUN | +258/-2 | 16 个文件

    With VLLM_USE_RUST_FRONTEND=1, allowed_token_ids, logit_bias keys, and /v1/completions token-id prompts are forwarded to the engine unchanged. An id at or beyond the model vocabulary makes the engine index the embedding table or the logits vector out of range, triggering a CUDA device-side assert that aborts EngineCore and takes down the whole server — a single well-formed request is a DoS. Th…

  • 1c3a72b #45180 — [Bugfix] Add fetch_images to MistralCommonImageProcessor (#45180)
    • 作者: Julien Denize | +81/-0 | 2 个文件

    Fix usage of mistral models going through MistralCommonImageProcessor that require fetch_images a recent method added by Transformers >= 5.10. Built upon https://github.com/vllm-project/vllm/pull/44989 Launched a ministral3 server and added unit test. ## Test Result all pass —

📖 Documentation

  • a37b4a9 #45301 — [Doc] AGENTS.md: add section about coding style (#45301)
    • 作者: Thomas Parnell | +9/-0 | 1 个文件

    Add instructions to AGENTS.md to steer agent-assisted contributions away from inserting excessively long comments and docstrings. Adapted from PyTorch’s CLAUDE.md: https://github.com/pytorch/pytorch/blob/main/CLAUDE.md#coding-style-guidelines n/a ## Test Result n/a —

  • c2b4cd3 #37047 — [Doc][Attention] Fix MLA top-of-file comments (#37047)
    • 作者: wineandchord | +10/-10 | 1 个文件

    Fix several obvious issues in the MLA top-of-file explanatory comment and pseudocode. ## Details - describe prefill as having a larger Sq / Skv ratio, close to 1 - describe decode as having a smaller Sq / Skv ratio, close to 0 - rename spda_o to sdpa_o in the pseudocode examples - fix the ql_nope pseudocode line to use q_nope - fix casual to causal in the chunked prefill pseudocode - remove the st…

🧪 CI/Tests

  • bd59c91 #45274 — [CI] ci-fetch-log.sh: fetch all failed jobs from a build URL or PR number (#45274)
    • 作者: Michael Goin | +174/-50 | 4 个文件

    Make CI log fetching one obvious command for humans and agents. Previously ci-fetch-log.sh required a job UUID, and web-UI URLs (/list?sid=<step_uuid>) failed with a 403 because the sid is a step uuid, not a job uuid. Now: Logs are saved as ci--.log with \t printed per job; - streams a single job to stdout; reruns keep existing files unless CI_FETCH_LOG_FORCE=1. Th…

  • 1ce3cdc #45302 — [ROCm][CI] fix fp8 support for test_deepep_moe (#45302)
    • 作者: Divakar Verma | +12/-9 | 1 个文件

    classic issue of float8_e4m3fn vs float8_e4m3fnuz Resolves CI tests for kernels/moe/test_deepep_moe.py::test_deep_ep_moe Buildkite link - link

⚡ Performance

  • 39dee11 #40660 — [MM][Perf][CG] Support ViT full cudagraphs for mllama4 (#40660)
    • 作者: allgather | +193/-14 | 5 个文件

    init commit has UT (unit test). FCG support is in mllama.py ## Testing 3 Tests: ut, e2e, benchmark eager vs cg FCG shows improvements across all stats except P99 tpot. Server: completion:

  • fcf5115 #44899 — [ROCm][DSv4][Perf] Flash-decode split-K decode attention kernel (#44899)
    • 作者: Fangzhou Ai | +675/-10 | 2 个文件

    This PR optimizes the DeepSeek-V4 sparse decode attention path on ROCm (_sparse_attn_decode_ragged_kernel), which is GPU-occupancy-bound at small batch sizes: DSV4’s 16 TP-local heads collapse to a single head-block, so the whole decode runs on just batch workgroups and leaves most of a 256-CU gfx950 device idle (latency flat at ~277us across batch 4 to 128). The fix rewrites it as a flash-dec…

🔧 Refactor

  • e0871ad #45104 — [Refactor] Chat Completions Streaming Harmony Refactor and Bugfixes (#45104)
    • 作者: yzong-rh | +394/-733 | 5 个文件

    Refactor Chat Completions Harmony paths to use unified Parser Before: - Harmony bypassed the unified Parser path. - Chat Completions non-streaming flowed through harmony_utils.py and openai_tool_parser.py. Two independent StreamableParser are created and output is parsed twice: once for reasoning/content and again for tool calls. - Chat completions streaming flowed through stream_harmony.py, which…

  • f712fd0 #45171 — [Refactor] Chat Completions Harmony Refactor, non-streaming path. (#45171)
    • 作者: yzong-rh | +822/-816 | 16 个文件

    Per reviewer request, extracted non-streaming path change from https://github.com/vllm-project/vllm/pull/45104. BFCL (which runs on the non-streaming path). ## Test Result For BFCL results, see #45104. Use AI assistance. cc @sfeng33 @bbrowning —

🖥️ Kernel

  • 2ec6594 #36902 — [Kernel][Helion][1/N] Add Helion kernel for per_token_group_fp8_quant (#36902)
    • 作者: Xiaohong (Sean) Chen | +4347/-4 | 10 个文件

    This PR is to add Helion kernel for operation. It follows the implementation from the vllm c version. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. Autotuning and benchmarking in H200 and B200 were contributed by @gmagogsfm. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: a…