共 50 个 commit，涉及 273 个文件，+10066/-5989 行变动。

概要

统计项 数值
Commit 数 50
变更文件 273
新增行数 +10066
删除行数 -5989

Commit 列表

📦 Other

  • ec0ffaa #47435 — [Rust Frontend] Improve scheduler stats logging parity (#47435)
    • 作者: Bugen Zhao | +521/-30 | 6 个文件

    Signed-off-by: Bugen Zhao i@bugenzhao.com This PR fills several straightforward Rust frontend periodic log-stats parity gaps with Python: - deferred requests, preemptions, and external prefix cache hit rate - spec decoding: accepted/draft throughput, accepted/drafted token counts, average draft acceptance rate, and per-position acceptance rate - mfu: existing estimated FLOPs/read/write counters …

  • 178fd56 #47410 — support GLM-5.2 gate use FP32 (#47410)
    • 作者: Yuxuan Zhang | +19/-0 | 2 个文件
  • 3e158ae #47428 — [ModelRunner V2] Fix Mamba2 crash on non-spec-decode (#47428)
    • 作者: Nick Hill | +1/-1 | 1 个文件

    MambaHybridModelState.build_metadata populated num_accepted_tokens on every forward pass, so the Mamba2 attention builder passed a wrongly shaped tensor (num_reqs, not num_decodes) straight into selective_state_update, tripping assert num_accepted_tokens.shape == (N,). This crashed hybrid Mamba2 + MoE models (e.g. GraniteMoeHybrid) now routed to V2. Gate the population on speculative decoding bein…

  • a2f7130 #44443 — [ModelRunner V2] Enable by default for all dense models (#44443)
    • 作者: Wentao Ye | +55/-17 | 6 个文件

    Part of https://github.com/vllm-project/vllm/issues/41286 Covered in CI

  • de2a8fc #47128 — [ROCm] [PyTorch] Move to stable abi since ROCm upgraded to torch 2.11 (#47128)
    • 作者: TJian | +22/-131 | 6 个文件

    Since ROCm has upgraded to torch 2.11 in PR https://github.com/vllm-project/vllm/pull/45362 , this PR follow up to clean up the previous fallback loop https://github.com/vllm-project/vllm/pull/44648 . So we are moving back to stable_abi. ## Test Result - pytest -svvvvv tests/kernels/core/test_uva.py - python3 -m pytest tests/lora/test_moe_lora_align_sum.py -q - Moe test —

  • 84b9c27 #47304 — Update DeepGEMM tag to point to latest nv-dev branch for sm120 support (#47304)
    • 作者: Michael Goin | +4/-2 | 2 个文件

    This PR https://github.com/vllm-project/vllm/pull/43477 landed in vLLM without bumping up our commit tag for DeepGEMM for the SM120 support, which resulted in many failures for models that trigger DeepGEMM kernels i.e. https://github.com/vllm-project/vllm/issues/47169, https://github.com/vllm-project/vllm/issues/47266 For now we should bump the DeepGEMM commit for nv-dev (which includes https://gi…

  • 25fcb65 #47283 — [Rust Frontend] Use enum-backed domain types for engine outputs and structured outputs (#47283)
    • 作者: Bugen Zhao | +760/-603 | 16 个文件

    Several engine-core protocol payloads arrive as product-shaped Python/msgpack structs, but Rust code wants finite semantic variants. Encoding those invariants in the public Rust types makes invalid states harder to construct and keeps validation closer to the serde boundary. Refactor some engine-core protocol types to expose enum-backed domain types northbound, while still perserving the existing …

  • 08a8a4a #46306 — feat(rust): expose profiler control routes in Rust frontend (#46306)
    • 作者: Pranav Thakur | +381/-2 | 11 个文件

    This adds the /start_profile and /stop_profile HTTP routes to the Rust frontend when profiling is enabled via –profiler-config. The routes forward the call to the Python engine’s profile utility RPC. Fixes #46092 Checks performed: - Duplicate PRs checked: None found addressing the Rust frontend profiler routes. - Tests run locally: - cargo test -p vllm-cmd –bin vllm-rs - cargo test -p vllm-serve…

  • b0b8a28 #44785 — [Model] Add LLaVA-OneVision-2 (LlavaOnevision2ForConditionalGeneration) (#44785)
    • 作者: chengzheng345 | +2593/-0 | 6 个文件

    Add support for LLaVA-OneVision-2 as a single multimodal architecture implemented in one model file (vllm/model_executor/models/llava_onevision2.py). Supported inputs: text + image + video (T + I+ + V+). - Image: AnyRes dynamic tiling + 2x2 spatial merge with exact placeholder accounting. - Video: three preprocessing backends selected through a single mm_processor_kwargs entry point (video_backend…

  • 09663ab #44977 — [ROCm][MLA] Fuse MLA q/kv RMSNorm + FP8 per-token quant in the FP8 attention path (#44977)
    • 作者: xaguilar-amd | +371/-5 | 4 个文件

    The MLADualRMSNormFusionPass fuses the paired q_a_layernorm + kv_a_layernorm of MLA attention into AITER’s fused_qk_rmsnorm HIP kernel, but that fusion silently stops firing once attention is FP8-quantized — the earlier RocmAiterRMSNormQuantFusionPass folds the q-side rms_norm → fp8 per-token quant into a single rocm_aiter_rmsnorm_fused_dynamic_quant op, so the q latent is no longer a bare rms…

  • 1360c42 #47319 — [UX] Include NVTX in cuda.txt (#47319)
    • 作者: Jee Jee Li | +2/-1 | 1 个文件

    Our code uses nvtx, so add it to cuda.txt, otherwise, when we set export VLLM_NVTX_SCOPES_FOR_PROFILING=1, the server will raise the following error. ## Test Result —

  • 2b753ad #47093 — [Spec Decode] DSpark speculators checkpoint support (#47093)
    • 作者: Michael Goin | +140/-16 | 4 个文件

    Validated with Qwen3-8B and GLM 5.2 DSpark (preview)

  • e196268 #47338 — [Docker] Remove unused Dockerfile.nightly_torch (#47338)
    • 作者: Andrey Talman | +0/-326 | 1 个文件

    Remove docker/Dockerfile.nightly_torch, which is dead/unreferenced. ## Why it’s safe Dockerfile.nightly_torch (added in #16936) is not referenced anywhere: - vLLM repo: no hits in .buildkite/ pipelines, docker/.hcl bake files, image_build.sh scripts, .github/workflows, or docs. - vllm-project/ci-infra: no references (ci.hcl, bake.hcl, bootstrap.sh, code search all empty). Building vLLM against t…

  • fa24813 #45723 — [MoE] Plumb gemm1_alpha/beta/clamp_limit into TRT-LLM FP8 MoE (#45723)
    • 作者: Yongye Zhu | +52/-3 | 4 个文件

    The FlashInfer FP8 block-scale MoE kernels — trtllm_fp8_block_scale_moe and trtllm_fp8_block_scale_routed_moe — accept three optional per-expert SwiGLU parameters (gemm1_alpha, gemm1_beta, gemm1_clamp_limit) that realize the OAI SwiGLU variant (X2 * sigmoid(alpha*X2) * (X1 + beta) with clamping) for MXFP8. The FP8 TRT-LLM experts did not pass them, so MXFP8 models using clamped/OAI SwiGLU (e.g. Mi…

  • d322943 #47229 — [DSV4] Better MXFP8 quantization kernel (#47229)
    • 作者: Yongye Zhu | +1/-0 | 1 个文件

    Apparently this cutedsl kernel is 2x of the original cuda kernel

  • 8cfeb84 #47308 — [ModelRunner V2] Warmup cross-attn properly in encoder-decoder case (#47308)
    • 作者: Nick Hill | +27/-2 | 1 个文件

    E.g. for whisper. Existing warmup was resulting in zero-key attention op.

  • 5fd4421 #46482 — [ROCm][P/D] MoRIIO toy proxy: support JSON Content-Type for OpenAI clients. (#46482)
    • 作者: Chaitanya Sri Krishna Lolla | +3/-0 | 1 个文件

    The Quart-based MoRIIO disaggregated toy proxy returns streamed responses with Quart’s defaults via make_response(stream_generator) without setting a Content-Type. Quart defaults a streamed response to text/html; charset=utf-8, so even on a successful 200 the body is mislabeled. Reference Documentation from Quart: This breaks OpenAI-compatible JSON clients. For example, lm_eval (local-completi…

  • f5a8d73 #46995 — [Spec Decode] DSpark (#46995)
    • 作者: Benjamin Chislett | +1821/-94 | 24 个文件

    Adds support for DSpark speculative decoding, using both DeepSeek-V4 DSpark models as well as DeepSeek-trained Qwen3-DSpark models. ## Design DSpark uses non-causal sliding-window attention. To implement this, instead of manually reimplementing the MLA attention, we instead utilize the existing SparseMLA backends with an expanded topk size, in which the queries all include each other in their to…

  • c638f92 #47265 — [Rust Frontend] Split engine core DTOs into separate modules (#47265)
    • 作者: Bugen Zhao | +806/-789 | 54 个文件

    Signed-off-by: Bugen Zhao i@bugenzhao.com Pure code structure refactoring without any logic changes, making the engine-core-client crate cleaner than mixing all definitions into mod.rs. Now we have: - request - output - sampling - structured_outputs This also helps if we want to isolate the northbound-facing shape from the wire representation of some type. Also clean up crate-root-level re-expor…

  • 13c49f9 #45368 — [xpu][lora]: Align LoRA implementation with Punica GPU: fix _apply_expand rank mismatch, add_inputs hardcode, and MoE EP (#45368)
    • 作者: Chaojun Zhang | +182/-35 | 3 个文件

    This PR fixes several bugs after #42290 and #41184 . ### Fixes 1. Rank dimension mismatch in _apply_expand for TP > 1 (non-fully-sharded LoRA) 2. Hardcoded add_inputs=True in add_lora_linear The add_inputs kwarg was popped but add_expand was always called with add_inputs=True, ignoring the caller’s value. This broke dual-stream LoRA which passes add_inputs=False. **3. add_inputs not forwar…

  • a78c156 #30966 — Migrate GPTBigCode and Starcoder2 to the Transformers modeling backend (#30966)
    • 作者: Harry Mellor | +4/-681 | 5 个文件

    Migrate support of GPTBigCode and Starcoder2 to use the Transformers modeling backend instead of having duplicated implementations in vLLM. These architectures are ~2 years old and do not feature in the top 100 most used architectures in vLLM. This migration: - Reduces the number of model implementations maintained in vLLM - Adds support for the following features that the Transformers modelling b…

  • 5c4db60 #45903 — docs(security): document gRPC interface as insecure for private use only (#45903)
    • 作者: Juan Pérez de Algaba | +21/-0 | 1 个文件

    Add a new section to the security guide documenting that the gRPC interface does not implement authentication, authorization, or encryption and must be protected via network-level access controls.

  • 4e5ca89 #47269 — [ROCm][MiniMax-M3] Cross-layer lightning-indexer top-k sharing (#47269)
    • 作者: Fangzhou Ai | +41/-1 | 1 个文件

    Adds opt-in cross-layer sharing of the lightning-indexer top-k block selection for MiniMax-M3 sparse attention. When use_index_cache is set, only 1 of every index_topk_freq sparse-attention layers recomputes the indexer score + top-k selection; the rest reuse the selection the preceding compute layer wrote into the shared topk_indices_buffer in the same forward pass. Adjacent sparse layers sel…

  • a22e0df #47263 — [Model] Remove AyaVision, MusicFlamingo (#47263)
    • 作者: Harry Mellor | +3/-1422 | 12 个文件

    Reasons: - Both of these architecture classes recieved very low vLLM usage - MusicFlamingo has been superseded by AudioFlamingo3 (new MusicFlamingo checkpoints use AudioFlamingo3) This PR removes both architecture classes and adds them to _PREVIOUSLY_SUPPORTED_MODELS

  • cc56379 #47192 — [Model] Support Hy3 token suffix and JSON Schema array types (#47192)
    • 作者: stevenkuang | +24/-11 | 2 个文件

    Support Hy3 token suffix and JSON Schema array types. Tests for the hy_v3 reasoning parser and tool parser.​ ## Test Result Passed — - Hy3 tokenizers append a suffix to the special tokens. Read the suffix from tokenizer.init_kwargs[“token_suffix”] and apply it to the parser’s special tokens. - Support JSON Schema array types, i.e. union-type declarations such as “type”: [“string”, “object”].

  • ed41aa2 #43950 — [ROCm][DSV4] Use aiter mHC pre/post as the default ROCm path (#43950)
    • 作者: Fangzhou Ai | +32/-39 | 3 个文件

    Re-enable the aiter multi-head-consensus (mHC) pre/post ops as the preferred ROCm path for DeepSeek V4. #43679 introduced the tilelang fused post+pre mHC kernel and explicitly left a hook to switch back to the (faster) aiter mHC kernels once an aiter release containing the mhc_pre_gemm_sqrsum_kernel race-condition fix was available. This is that follow-up. ### Dispatch / fallback path Selectio…

  • 77a9c5a #44353 — Weight sync refactor + move sparse nccl engine (#44353)
    • 作者: Aaron Hao | +953/-1046 | 32 个文件

    Current state of the weight sync path requires non weight transfer engine components to assume too much about weight sync internals. Gpu workers should not need to worry about whether to run layerwise reload, or how to construct weight loaders for weight transfer engines. These concepts are weight transfer engine specific, and should be taken care of by the engine. Also, NCCL sparse weight update …

  • f651a8a #42486 — [XPU][UT]Enable ut qk_norm_rope_fusion (#42486)
    • 作者: Yejing Lai | +10/-7 | 2 个文件

    Removed ut qk_norm_rope_fusion cuda hard code, added xpu support.

  • 8f82be5 #47242 — [CI/Build] Fix LoRA testing (#47242)
    • 作者: Jee Jee Li | +17/-2 | 1 个文件

    Test Result —

  • a461070 #47243 — [Core] Make sleep-mode backend capability flags communicator-agnostic (#47243)
    • 作者: Nils Matteson | +15/-14 | 2 个文件

    Follow-up to #44074. Renames the two SleepModeBackend capability flags that named NCCL specifically, so the interface stays generic across communicator backends: - preserves_nccl → preserves_communicators - preserves_graphs_with_nccl → preserves_graphs_with_communicators This addresses @galletas1712’s review feedback on #44074 (NVIDIA Dynamo team): NCCL is not the only comms backend in vLLM, so a …

  • 4470ae8 #46806 — Remove mantis (#46806)
    • 作者: Tiezhen WANG | +8/-203 | 10 个文件

    We are running a sprint to reduce the technical debt of the vLLM code base starting by removing obsolete model architectures based on vLLM internal usage tracking. The usage of both Grok models is very low, less than 20 hours in the past 3 months. Confirmed with Prof. Wenhu Chen from Tiger research ## Test Result —

🐛 Bug Fix

  • a47f38f #47383 — [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels (#47383)
    • 作者: Woosuk Kwon | +148/-4 | 2 个文件

    _compute_cumulative_log_p_kernel and _compute_local_residual_mass_kernel (used when rejection_sample_method=“block”) multiply int32 logit/request-state indices by vocab-scale strides before int64 promotion. With a GLM-scale vocab (~155k tokens) the byte offset wraps once logit_idx * vocab >= 2**31 (logit_idx >= ~13.8k, reachable with large max_num_seqs × 1 + num_speculative_tokens), or once req_st…

  • 2665ed7 #46838 — [Bugfix][Kernel] Correct FlashInfer CUTLASS MoE tuning token bound (#46838)
    • 作者: Hiki | +0/-2 | 1 个文件

    Previously, #34542 refactored the FlashInfer CUTLASS fused MoE backend and passed the CUDA graph max capture size (typically 512) as tune_max_num_tokens for flashinfer_cutlass_fused_moe. However, max_capture_size describes graph capture capacity and can be unrelated to the scheduler/MoE max batched token limit, so using it can select a tuning configuration for the wrong token range and thus cause …

  • d63c8e9 #47238 — [BugFix][Spec Decode] Compact shared topk indices buffer after first MTP draft step (#47238)
    • 作者: Giancarlo Delfin | +27/-0 | 3 个文件

    Context Deepseek-style topk index selection for MTP has a bug during proposal stage when the topk indices are shared among all MTP draft step forward passes. After the first draft step, set_skip_topk is called on the MTP model to force the remaining draft steps to reuse the same topk indices values (held within topk_indices_buffer of the attention module). The issue is that the first draft forwa…

  • 7fe7fa9 #47208 — [CI][Bugfix] Rerun test_engine_log_metrics_ray on Ray GCS startup timeout (#47208)
    • 作者: peizhang56 | +13/-0 | 1 个文件

    tests/v1/metrics/test_ray_metrics.py::test_engine_log_metrics_ray intermittently fails on ROCm CI (e.g. buildkite build 75329, the “AMD: V1 Core + KV + Metrics” step) with: The test starts a Ray cluster implicitly: the first EngineTestActor.remote() call triggers Ray’s auto_init_ray() FR bare ray.init(). On a contended CI host, Ray’s GCS server subprocess occasionally fails to write its port fil…

  • 4787f2d #47305 — [Bugfix] Don’t read KV cache past seq_len in triton paged attn kernels (#47305)
    • 作者: Nick Hill | +13/-6 | 2 个文件

    The Triton unified_attention and ROCm paged-decode kernels load KV cache slots beyond seq_len (the unwritten tail of the last partial block) and rely on the score mask to drop them. That is unsafe when the tail is non-finite: masked positions get softmax weight 0, but 0 * NaN = NaN still poisons the output. In compute_tile_loop_bounds the causal-derived max_seq_prefix_len can additionally overshoo…

  • 00eb7ce #47029 — [Bugfix] Prevent padding placeholders from reaching embeddings (#47029)
    • 作者: Yiliu Dong | +60/-0 | 3 个文件

    Fix a CUDA device-side assert that can occur when speculative padding placeholders are scheduled as draft tokens. Scheduler-side speculative padding uses -1 draft token IDs to keep the first decode step at the uniform 1 + num_spec_tokens shape. Those placeholder IDs should be visible to rejection sampling so they are never accepted, but they must not be fed into the model embedding lookup. ## Envi…

  • c8bdcc0 #47135 — [Bench][BugFix] Fix empty decoder prompt for Cohere ASR in throughput benchmark (#47135)
    • 作者: Michał Ganczarenko | +85/-1 | 2 个文件

    Fix ASRDataset.sample() raising “The decoder prompt cannot be empty” when benchmarking Cohere ASR models (e.g. CohereLabs/cohere-transcribe-03-2026). ASRDataset.sample() previously only built a decoder prompt for OpenAI/Whisper tokenizers, falling back to prompt = "" for all others. CohereAsrTokenizer sets skip_decoder_start_token=True and does not override skip_prompt_length_check, so the empty p…

  • 63fcce4 #47031 — [Bugfix] Fix GraniteMoeShared weight loading broken by #41184 (#47031)
    • 作者: Michał Ganczarenko | +3/-3 | 1 个文件

    Fix KeyError when loading GraniteMoeSharedForCausalLM weights after the FusedMoE/MoERunner inversion refactor (#41184). After #41184, the parent _load_weights() expects expert keys of the form experts.{e}.w{n}.weight. GraniteMoeShared.load_weights() was still inserting routed_experts before delegating, producing a doubled path experts.routed_experts.routed_experts.{e}.w{n}.weight and a KeyError. R…

  • 024b06b #42748 — [Bugfix] Expose usage field in GenerateResponse for disaggregated serving (#42748)
    • 作者: Aleksei Ivashov | +4/-2 | 2 个文件

    serving.py already constructs UsageInfo (including prompt_tokens_details.cached_tokens when –enable-prompt-tokens-details is set) and passes it to the GenerateResponse constructor. However, the usage field was missing from the GenerateResponse model, so pydantic silently dropped it. Fix: add usage: UsageInfo | None = None to GenerateResponse, mirroring GenerateStreamResponse which already has…

  • aa8bb55 #46730 — [ROCm][Perf][Bugfix] DSv4 indexer: use platform FP8 dtype (fnuz) for Q-quant on gfx942 (#46730)
    • 作者: akii96 | +19/-8 | 1 个文件

    On gfx942 the DeepSeek-V4 Flash indexer quantizes Q and K to different FP8 types. K already uses the platform type (e4m3fnuz on gfx942, via current_platform.fp8_dtype()), but the fused RoPE+quant kernel in fused_indexer_q.py hardcodes Q to e4m3fn. The FP8 logits kernel then gets fnuz K with fn Q and falls back to a mixed-dtype path on every call. This change derives Q’s type from current_platform….

  • fa4bec9 #47071 — [Bugfix] Fix pooled Whisper sliding-window KV sizing (#47071)
    • 作者: Andy Lo | +43/-2 | 2 个文件

    Voxtral Realtime’s causal Whisper encoder uses block_pool_size encoder tokens per pooled KV block, but its SlidingWindowSpec.sliding_window was still expressed in encoder-token units. The sliding-window KV cache manager then reserved about block_pool_size times too many blocks for the encoder cache. The fix converts the KV-cache spec’s sliding window into pooled units while leaving the attention k…

  • 697c34b #47126 — [Bugfix] Fix beam search candidate indexing when logprobs count varies (#47126)
    • 作者: Chauncey | +119/-51 | 2 个文件

    [Bugfix] Fix beam search candidate indexing when logprobs count varies see https://buildkite.com/vllm/ci/builds/75148#019f1408-624f-485c-a3ea-762444b94b6a before: ## Test Result —

✨ New Feature

  • 3af8789 #38174 — [Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) (#38174)
    • 作者: Wonderful | +364/-5 | 8 个文件

    Implements Token-Level Intersection (TLI) speculative decoding, allowing target and draft models to have different (but overlapping) vocabularies. Closes #38173 ## Algorithm Based on the ICML 2025 oral paper: > Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies > Timor et al., https://arxiv.org/abs/2502.05202 How it works: 1. At startu…

🧪 CI/Tests

  • 8357226 #47376 — [XPU][CI] Split test_punica_ops into separate pytest invocations for stability (#47376)
    • 作者: Chaojun Zhang | +3/-1 | 1 个文件

    Stabilize the XPU LoRA Punica kernel CI by splitting test_punica_ops.py into three separate pytest invocations, rather than running the full suite in a single process. This avoids the test failures that occur when all tests are executed together, and removes the need for an ever‑growing list of –deselect exclusions.

  • e91f5f8 #47342 — [CI] Remove torch_nightly mirror tags (superseded by TORCH_NIGHTLY full-nightly build) (#47342)
    • 作者: Andrey Talman | +0/-44 | 3 个文件

    Remove the torch_nightly step tags that drive the curated “vLLM Against PyTorch Nightly” mirror group. This mirror mechanism selects a subset of steps (tagged torch_nightly: true / mirror: { torch_nightly: {} }) that the ci-infra pipeline generator collects into a separate “vLLM Against PyTorch Nightly” group and runs against the torch-nightly image under NIGHTLY=1. It is superseded by the *…

  • f1cf6b0 #47299 — [CI] Fix segfault in tracing test (#47299)
    • 作者: Nick Hill | +4/-5 | 1 个文件

    This was actually already fixed for rocm case but it is not rocm-specific and the regular metrics test fails intermittently due to this. E.g. https://buildkite.com/vllm/ci/builds/75740/list?jid=019f1d4f-1e55-4e15-b6db-fd8f495f8bef&tab=output

  • e7d0fcb #47197 — [CI] Fix various failures on main (#47197)
    • 作者: Harry Mellor | +18/-14 | 5 个文件
  • dee5da1 #47250 — [Test] Run SageMaker handler-override tests in-process via TestClient (#47250)
    • 作者: Jyothirmai Kottu | +227/-333 | 1 个文件

    The SageMaker handler-override integration tests in tests/entrypoints/serve/sagemaker/test_sagemaker_handler_overrides.py each launched a full model server via RemoteOpenAIServer, only to exercise customer handlers that fully replace the /ping and /invocations endpoints. No inference engine is needed to validate override behavior. Beyond being slow, starting a server made these tests sensitive t…

🔩 Misc

  • d0a2584 #46984 — [Misc] Use functions instead of PTX for the PDL instruction (#46984)
    • 作者: Jee Jee Li | +27/-25 | 7 个文件

    Test Result —