共 65 个 commit，涉及 422 个文件，+21819/-5893 行变动。

概要

统计项 数值
Commit 数 65
变更文件 422
新增行数 +21819
删除行数 -5893

Commit 列表

📦 Other

  • 837eae6 #50298 — [DSv4 Perf] Remove redundant full kernel for dsv4, 1.88x kernel performance improvement (#50298)
    • 作者: Wentao Ye | +44/-23 | 3 个文件

    Part of https://github.com/vllm-project/vllm/issues/45861 Passing an out tensor to avoid additional torch.full kernel call Covered in unit tests Perf can be seen in this AI generated script And we get

  • 904fae8 #50312 — [DSv4 Perf] Fix redundant memory allocation and copy for dsv4 pp buffer, 448 MiB GPU memory saved (#50312)
    • 作者: Wentao Ye | +24/-28 | 3 个文件

    We do torch.empty and copy in pp last rank even if there is no mtp enabled, this PR fixes the issue, Perf gain can be seen in this AI generated script And we get

  • 5b95890 #50339 — [FlexAttention] Avoid encoder block-mask compile explosion (#50339)
    • 作者: Andreas Karatzas | +66/-13 | 4 个文件
    • Default encoder-only FlexAttention masks to 128-token Q/KV blocks, while keeping the existing small-block defaults for paged KV attention and honoring explicit block-size overrides. The long-text fixture also uses a local seeded generator so token counts and compiler shapes are reproducible. - On MI355, the exact cold-cache test improved from 300.31 seconds to 37.99 seconds; the full long-text m…
  • 30b4e7f #48981 — [rl] Stateful Trainer Send: IPC [2/N] (#48981)
    • 作者: Aaron Hao | +392/-422 | 8 个文件

    Context Second of the planned three-PR split of the trainer-side weight-transfer rework. - PR 1 (merged): introduced the new trainer-side abstractions (WeightSource / ModuleSource, VLLMWeightSyncClient, TrainerWeightTransferEngine, WeightTransferTrainerFactory). Purely additive; no backend migrated. https://github.com/vllm-project/vllm/pull/48042 - PR 2 (this one): migrate the IPC b…

  • 59e831c #48757 — [Compilation]Fuse Transformers Residual Add + RMSNorm (#48757)
    • 作者: BadrBasowid | +298/-2 | 4 个文件

    Transformers models emit residual add and RMSNorm separately, preventing the use of FusedAddRMSNorm kernels. This change canonicalizes them into fused_add_rms_norm and handles the intervening reshape. Furthermore, the separate residual add and reshape operations prevented some patterns, like AR + rms, from being discovered and fused. Since this PR hides those offending ops, the fusions are fixed a…

  • 1a20d23 #48947 — [PARSER][Mistral] unified engine-based parser for reasoning and tool calls (#48947)
    • 作者: Julien Denize | +2931/-1492 | 30 个文件

    This PR aims to discard previously separated tool call and reasoning parsers to a unified one. It should supports all mistral format tokenizers including: - pre v11 version (old tool calling format and no reasoning) - post v11 version (new tool calling format and potentially reasoning) - text or special token based reasoning - grammar from mistral-common guidance - empty tool-call names are now va…

  • e2efe79 #47207 — [ROCm]Migrating Deepseek V3.2 to vllm/models/deepseek_v32/ (#47207)
    • 作者: sroberts-amd | +1031/-70 | 12 个文件

    This PR migrates deepseek_v32 to /vllm/models/deepseek_v32 This is an effort done with the help of Claude and is cleaned up through human review. Migrate the AMD specific needs for deepseekv32 to vllm/models/deepseek_v32/ Following patterns of other models, moved common elements into common, updated nvidia dependencies to reflect move. Validate DeepseekV3 2 > VLLM_ROCM_USE_AITER=1 vllm serve deeps…

  • 38a267c #49570 — [MyPy][1/N] Fix mypy errors in some tests/ directories and enforce follow-imports=silent (#49570)
    • 作者: Martin Hickey | +166/-101 | 30 个文件

    Part 1 of enabling mypy for tests directory. Implements PR0 and PR1 of the plan defined in feature #49569. MyPy checks tests/** with –follow-imports skip which hides real type errors. group_files() also claims every tests/** file for the “tests” entry in SEPARATE_GROUPS regardless of more specific entries also being present. Therefore removing a directory from SEPARATE_GROUPS alone does not enfor…

  • 89d97d9 #50397 — docs(security): document Ray cluster trust model and env var propagation (#50397)
    • 作者: Juan Pérez de Algaba | +98/-0 | 1 个文件

    Adds a new section to the security guide explaining that vLLM treats the entire Ray cluster as a single trust domain and that RayExecutorV2 propagates driver environment variables to workers by default. Includes hardening recommendations for operators who scope credentials to the driver alone.

  • 072a472 #50387 — [CPU] Bump up CPU kernels to latest version (#50387)
    • 作者: Li, Jiang | +2183/-1292 | 15 个文件

    Sync csrc/cpu/sgl-kernels/ from sglang (sgl-kernel/csrc/cpu) to latest upstream main, reconciling vLLM-specific patches layered on since the last bump (#41924): MXFP4 W4A16 MoE, AMX GDN dispatch, ISA-portability BLAS fallback, RISC-V scalar/RVV support, chunked-prefill has_initial_state fix, DFlash speculative decoding, and batched spec-decode conv-state updates. Also rejects one upstream regressi…

  • aeeb36b #50000 — [New model] Kimi K3 (#50000)
    • 作者: Jiangyun Zhu | +2931/-1349 | 82 个文件

    7.30 Note: after merging, it still needs to install https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16rc5 to run K3 add moonshotai/Kimi-K3 model support Please use the docker image vllm/vllm-openai:kimi-k3 to run it, see vllm recipes https://recipes.vllm.ai/moonshotai/Kimi-K3. This branch has some extra dependencies: - SITU trtllmgen MOE: https://github.com/flashinfer-ai/flashinf…

  • f1e8fd2 #49937 — [ROCm] Add AITER FP8 ViT encoder attention (#49937)
    • 作者: yinfengLiu | +563/-45 | 8 个文件
    • enable AITER per-tensor FP8 attention for multimodal vision encoders on gfx942 and gfx950 - preserve packed variable-length image/video boundaries through AITER’s native varlen kernel - add compile-safe custom-op wrapping, architecture/symbol validation, correctness tests, a benchmark, and ROCm usage docs ## Why this is not duplicate work I searched open vLLM PRs for AITER FP8 ViT encoder attent…
  • a7a204c #50322 — Add FlashMLA H100 tests to CI, fix them after #32810 (#50322)
    • 作者: Jane (Yuan) Xu | +44/-6 | 3 个文件

    I recently ran the H100 tests locally for flashmla and realized that some were broken after the interface changed in #32810. I fixed them, and to prevent further regressions, I added them to CI. I haven’t worked with buildkite before so I’m not sure it’s correct, but I did it with the help of Claude. cc @LucasWilkinson who authored #32810 and @Harry-Chen CI, specifically these three test files sho…

  • 1ad5182 #50357 — [CI/Build] Limit wheel size check to CUDA 13 (#50357)
    • 作者: Tyler Michael Smith | +4/-6 | 3 个文件

    Fix wheel size check

  • e5f48df #47124 — [Quantization][Autoround][XPU] Add W4A16(moe) / MXFP4(linear/moe) Support (#47124)
    • 作者: lkk | +816/-48 | 9 个文件

    Supports deployment of AutoRound-format quantized models on XPU, including both dense and MoE architectures, with W4A16 and MXFP4 quantization schemes. ### W4A16 dense on XPU vllm ({‘pretrained’: ‘./Qwen3-8B_autoround_w4a16_rtn_ar/Qwen3-8B-w4g128/’, ’tensor_parallel_size’: 4, ‘max_model_len’: 8192, ‘max_num_batched_tokens’: 32768, ‘max_num_seqs’: 128, ‘add_bos_token’: True, ‘gpu_memory_utilization…

  • b28c178 #38293 — Fix: FusedMoE AssertionError with Speculative Decoding on Quark-Quantized Models (#38293)
    • 作者: Venkateswara Rao Cherukuri | +0/-0 | 0 个文件

    Root Cause When using speculative decoding (MTP) with Quark-quantized MoE models (e.g. amd/DeepSeek-R1-MXFP4), the vLLM server crashes during model loading with: The Quark config for this model excludes re:model.layers.61.* from quantization. The MTP drafter’s MoE layer at model.layers.61.mlp.experts matches this exclude pattern. QuarkConfig.get_quant_method() then returns UnquantizedLinearMet…

  • 1cb3fe5 #50329 — [CI Bugfix] Temp disable Humming wNa8 INT8 H100 CI (#50329)
    • 作者: Michael Goin | +1/-1 | 1 个文件

    There is a known failure in humming with int8 activations on Hopper, so disabling that test for now. Repro https://gist.github.com/HDCharles/6c65dca0953d4ec7a0d835d26126ff66 Latest nightly failure https://buildkite.com/vllm/ci/builds/80964/list?sid=019fac77-4414-4f3e-a076-1fd24c232d5b&tab=output ## Test Result —

  • 5fa0154 #49647 — [Rubin] Enable NVLink all-reduce paths on SM107 (#49647)
    • 作者: zaristei | +23/-0 | 3 个文件

    Part of the ongoing effort to bring up Rubin (sm_107) support in vLLM (similar to #49387). sm_107 has no entries in vLLM’s collective-communication selection tables, so the optimized NVLink all-reduce paths are all skipped and traffic silently falls back to NCCL. This PR wires sm_107 into those paths: - Custom all-reduce — enables the one-shot/two-shot NVLink kernels - **PyTorch symmetric-…

  • 48fc2e2 #50033 — feat(grpc): add KV event source discovery (#50033)
    • 作者: Connor Carpenter | +179/-5 | 13 个文件
    • Add Control.GetKvEventSources for discovering configured ZMQ KV-event publishers. - Return one source per data-parallel rank, including the resolved event and replay endpoints, topic, encoding, schema version, and queue settings. - Expose the publisher’s resolved runtime configuration through the EngineCore ready handshake so the Rust frontend does not duplicate rank-based endpoint resolution. -…
  • 9347745 #50244 — [torch.compile] Compile CustomOp.forward_native for ReLU^2 to avoid raw torch ops inside opaque custom ops (#50244)
    • 作者: roikoren755 | +2/-2 | 1 个文件

    Following in the footsteps of #32806 , this PR enables the same compile_native flag for the ReLU^2 activation, used by NemotronH models. Before: The first kernel is an elementwise clamp scalar kernel, which takes around 1.47us. The second an elementwise pow tensor scalar, taking around 1.34us. After: A single fused triton kernel, performing pow and ReLU, taking around 1.5us. For nvidia/Nemotron-3-…

  • 5c7a7f9 #49066 — [docs] Add documentation for pynvvideocodec video decoding backend (#49066)
    • 作者: Brandon Pelfrey | +56/-10 | 1 个文件

    Follow-up to contributions in https://github.com/vllm-project/vllm/actions/runs/27565608286 released in v0.25.0 pynvvideocodec has been recently merged but documentation is missing for the site. These added docs help users understand how to invoke the backend. CI docs build —

  • f98061c #48883 — fix(step3p5-mtp): honor exclude_modules for the MTP head via prefix (#48883)
    • 作者: Chanh Nguyen | +11/-2 | 1 个文件

    Problem MTP (–speculative-config ‘{“method”:“mtp”,…}’) crashes at model load for checkpoints that keep the MTP (nextn) layers unquantized on a quantized main model — e.g. stepfun-ai/Step-3.7-Flash-NVFP4: ## Root cause The checkpoint’s modelopt exclude_modules lists the MTP layers + lm_head (kept BF16). vLLM’s modelopt get_quant_method(layer, prefix) decides exclusion by fnmatch-ing a layer’s…

  • 8255369 #50153 — [KV Connector] Fix NIXL mamba state pairing for multi-slot block tables (#50153)
    • 作者: Nick Hill | +163/-34 | 6 个文件

    The NIXL P/D transfer pairs SSM state blocks positionally and assumes a request’s mamba group resolves to a single local state block. That only holds for the mamba_block_size = max_model_len default WITHOUT speculative decoding. With a speculative config, mamba managers co-allocate num_speculative_blocks trailing scratch slots on every request, and with mamba prefix caching the block list is addit…

  • 43eaefb #48791 — [ModelRunner V2] Enable sequence pooling for embedding and classification models (#48791)
    • 作者: Taneem Ibrahim | +509/-37 | 10 个文件

    Fixes the sequence-level portion of https://github.com/vllm-project/vllm/issues/41286 and unblocks that portion of https://github.com/vllm-project/vllm/pull/48290. After https://github.com/vllm-project/vllm/pull/49331 added the encoder-only attention and model-state foundation, Model Runner V2 still did not execute a model’s pooler: it hardcoded last-token normalization, assumed every pooling mode…

  • 242c591 #49341 — [Rust Frontend] Send multimodal tensors in auxiliary frames (#49341)
    • 作者: Reid | +530/-159 | 11 个文件

    Rust frontend EngineCoreRequest::Add messages currently convert multimodal tensors into owned raw bytes, copy those bytes into one large msgpack payload, and copy that payload again into the zeromq send buffer. Large images, multiple images, and video requests can therefore require several full-size memory copies and one large contiguous msgpack allocation. Although the wire representation already…

  • 72297d8 #47121 — [XPU] Route weightless RMSNorm to _C dispatch (#47121)
    • 作者: Yintong Lu | +0/-6 | 1 个文件

    Background After vllm-xpu-kernels adds support for weightless RMSNorm, vLLM dispatch should no longer send XPU weight=None calls to native fallback paths. Currently, if dispatch still falls back (for weight=None), the new XPU kernel capability is not fully utilized. ### Problem - Kernel capability exists in vllm-xpu-kernels. - vLLM dispatch path may still use native/IR fallback for XPU weight=…

  • 542a8fa #50094 — [KV Offload] Move CPUOffloadingSpec onto SharedOffloadRegion (#50094)
    • 作者: Chang Guo | +132/-5 | 2 个文件

    Move the default KV-offload backend CPUOffloadingSpec’s worker-side CPU buffer from a per-rank private pinned torch tensor onto the existing shared SharedOffloadRegion mmap, on CUDA/ROCm. This is the allocation-swap prerequisite for the TP-deduplication feature requested in #47929; it introduces no deduplication semantics on its own. Maintainer authorization (upstream reviewers do not see the …

  • 9a4e5f9 #43538 — [CI/Perf] Fix malformed serving benchmark config (#43538)
    • 作者: Minh Vu | +0/-5 | 1 个文件

    Fixes #43537. This removes a stray duplicate object from .buildkite/performance-benchmarks/tests/serving-tests.json that makes the default serving benchmark config invalid JSON. ## Root cause PR #43262 converted the config to the newer defaults-based format but left an extra dataset_name / dataset_path object after serving_llama8B_tp1_sharegpt. ## Validation Ran locally: The JSON parses successful…

  • c44e191 #49604 — [Rust Frontend] Add –limit-mm-per-prompt support (#49604)
    • 作者: Maria Guevara | +413/-16 | 14 个文件

    Adds –limit-mm-per-prompt as an optional CLI flag, capping how many input items of each modality (image, image_embeds, audio, video) appear per chat request. Modalities absent from the map default to unlimited. Rejects with HTTP 400 in validate_mm_limits, before fetch_media runs. This matches the python behavior, which validates item counts per modality against limit_per_prompt in vllm/config/mul…

  • dad7a63 #49582 — [EC Connector] Add has_pending_push_work (#49582)
    • 作者: omerpaz95 | +16/-0 | 2 个文件

    Adds has_pending_push_work() to ECConnectorBase, mirroring the existing KV connector API (vllm/distributed/kv_transfer/kv_connector/v1/base.py). This gives EC connectors a hook to signal that push-mode transfer work is still in flight, and wires it into Scheduler.has_requests() so the engine main loop keeps stepping until any pending EC push work drains. Default implementation returns False, so co…

  • 5b29c95 #48677 — [XPU] upgrade to torch 2.13 (#48677)
    • 作者: Yan Ma | +35/-92 | 7 个文件

    This PR upgrades to torch 2.13 on XPU platform and bump up v0.1.12 vllm-xpu-kernels. Note that dev environment is not installed. ## Test Result —

  • ad5d29d #50210 — [Model] Support Qwen3.5 text-only dense and MoE models (#50210)
    • 作者: Perkz Zheng | +62/-0 | 5 个文件
    • Register Qwen3_5ForCausalLM and Qwen3_5MoeForCausalLM as text-generation architectures. - Register the qwen3_5_text and qwen3_5_moe_text Hugging Face configuration types. - Mark the causal-LM base as hybrid and provide the GDN state dtype, shape, and copy hooks required by hybrid scheduling. - Reuse the existing Qwen3.5 cache-dtype handling and remove multimodal M-RoPE fields for text-only causa…
  • f5a7cce #50093 — [Model] Add Kimi K3 support: Python frontend [2/2] (#50093)
    • 作者: Bugen Zhao | +3056/-5 | 20 个文件

    Extract the Kimi K3 Python frontend from #50000 as the top layer of the independent frontend stack rooted at main through #50104. This layer adds Python XTML rendering, reasoning and tool parsing, structural-tag handling, and streaming support. Model-side preprocess_messages cleanup lives in #50089. This PR contains no vllm/models/kimi_k3 or rust/ changes. An open-PR search found this as the sole …

  • 65a1a16 #50194 — [CPU] Fix FP8 attention scratchpad sizing (#50194)
    • 作者: Tianmu Li | +35/-7 | 5 个文件

    The CPU attention scheduler computed FP8 KV tile geometry using sizeof(kv_cache_t). With BF16 queries and one-byte FP8 KV cache entries, this selected larger tiles than the BF16-backed scratchpad can hold during large AMX prefills. The resulting out-of-bounds writes cause corrupted output Unit Model ## Test Result | Metric | Before | After | |—|—:|—:| | Strict / flexible | 0.67 / 0.67 | 0.87…

🧪 CI/Tests

  • f388dd6 #50447 — [XPU][CI] skip kimi-k3 test (#50447)
    • 作者: Kunshang Ji | +1/-0 | 1 个文件

    fix CI by skip unsupported kimi-k3 model ## Test Result —

  • 0c64be8 #49839 — [Test][ROCm] Account for gfx950 FP8 RMSNorm rounding (#49839)
    • 作者: Andreas Karatzas | +35/-10 | 2 个文件
    • Use the shared FP8 allclose contract only for the measured gfx950 fused RMSNorm cases. - Bound every accepted difference to one FP8 ULP while leaving scales, residuals, other dtypes, other architectures, and other ROCm checks unchanged. E4M3 adjacent codes can differ by 12.5%, and its minimum subnormal step is approximately 0.002. Fused and unfused gfx950 reductions selected adjacent codes at th…
  • 61c1d09 #50284 — [CI] Stabilize speculator memory teardown (#50284)
    • 作者: Andreas Karatzas | +37/-29 | 1 个文件

    Use managed runners for the speculator and reference engines so ROCm teardown completes before the next engine starts. Motivation: - https://buildkite.com/vllm/amd-ci/builds/11417/list?sid=019fad1a-516e-49db-9547-c075d31390bb&tab=output

  • 165ed33 #50340 — [CI][ROCm] Stabilize LLM GC teardown check (#50340)
    • 作者: Andreas Karatzas | +9/-0 | 1 个文件

    Retry only the intermittent weakref assertion on ROCm after fixture cleanup without forcing cyclic collection and weakening cycle detection.

  • 48a077e #50414 — [CI] Improve comment-triggered authorization and retries (#50414)
    • 作者: Kevin H. Luu | +363/-17 | 4 个文件

    Improve the /ci run and /ci retry workflow so CI access is explicit, reviewable, and useful before an entire Buildkite build finishes. ### Authorization - Users with repository write, maintain, or admin permission can always run CI on an open PR, regardless of authorship. - Configured CI_TRUSTED_USERS can also always run CI. - Other contributors can run CI only on their own non-draft PR after eith…

  • 445d3aa #50318 — [CI] Retry failed steps on new PR commits (#50318)
    • 作者: Kevin H. Luu | +376/-26 | 2 个文件

    Extend the existing /ci retry command across PR commits. - If the current PR head already has a Buildkite build, preserve the existing same-build failed-job retry behavior. - If the current head has no build, find the latest completed command-triggered build for the same PR, fetch its latest failed/timed-out/expired script jobs, deduplicate their stable step_key values, and create a filtered build…

  • 0028fc8 #50207 — [XPU][CI]Add back skipped V1 test (#50207)
    • 作者: xiangdong | +2/-2 | 2 个文件

    Add back tests/v1/e2e/general/test_correctness_sliding_window.py::test_sliding_window_retrieval[True-1-5-google/gemma-3-1b-it] ## Test Result —

  • b889166 #48257 — [ROCm] [CI] Support cached K/V (key/value=None) in Triton prefix-prefill (#48257)
    • 作者: stefankoncarevic | +525/-40 | 2 个文件

    On ROCm, some models re-attend an already-cached sequence with the query only, calling attention with key=None / value=None. A concrete example is IQuest LoopCoder, whose loop layer calls global_attn(q, None, None): key=None / value=None is an existing contract in vLLM attention (used for cross-attention / cached re-attention and already documented in cpu_attn.py and rocm_aiter_fa.py). With aiter …

  • fa2a258 #50304 — [CI][ROCm] Fix AMD nightly distributed regressions (#50304)
    • 作者: Andreas Karatzas | +9/-117 | 2 个文件
    • PR #50089 added custom all-gather/reduce-scatter tests before the matching CustomAllreduce Python APIs landed, causing all seven cases in test_custom_all_reduce.py to fail. - Revert only the 117 premature AG/RS test lines while preserving the four existing ROCm custom-all-reduce cases that passed in the previous nightly. - The missing AG/RS integration remains in the open Kimi K3 umbrella PR #50…
  • 435c4da #50311 — [ROCm][CI] Avoid Ray worker startup env race (#50311)
    • 作者: Andreas Karatzas | +2/-2 | 2 个文件
    • No vLLM PR introduced this failure: the good and failing builds used the same Ray 2.54.0 pin and byte-identical Ray executor and pipeline-test code. - The vulnerable async metrics-exporter initialization came from Ray #55152; the getenv/setenv race was fixed by Ray #61034 and Ray #61281. - Ray 2.55.0 is the first release containing the fix, and ROCm CI is pinned to its 2.55.1 patch release. - Th…
  • 381b691 #50262 — [ROCm][CI] Fix Kimi K3 KDA on ROCm (#50262)
    • 作者: stefankoncarevic | +8/-2 | 2 个文件

    The AMD “Kernels KDA Test” job (mi300_1) was red. Investigation uncovered two separate problems, fixed here in two commits: 1. Stale CI path the job pointed at a test file that was moved by another PR, so it failed before running anything. 2. ROCm false-positive kernel selection once the job was repointed and actually ran, it surfaced a genuine ROCm bug: FlashKDA (a CUDA-only extension) wa…

  • 625871b #50241 — [CI][Test] Fix pooling truncation test after VLLMError hierarchy change (#50241)
    • 作者: stefankoncarevic | +3/-1 | 1 个文件

    tests/models/language/pooling/test_truncation_control.py::test_bigger_truncation_size began failing on main. The test asserts pytest.raises(ValueError) when truncate_prompt_tokens > max_model_len, but the request-validation path now raises VLLMValidationError, which is no longer a subclass of ValueError, so the exception is not caught and the test fails. This updates the test to expect VLLMValidat…

  • 5b14019 #50222 — [CI] Fix MXFP8 MOE backend selection tests on gfx942 (#50222)
    • 作者: fxmarty-amd | +11/-7 | 1 个文件

    As reported by @jikunshang (https://github.com/vllm-project/vllm/pull/49747#issuecomment-5114449319), #49747 includes two new failing tests, fixing in this PR. These failures were overlooked as local testing of #49747 was done on MI350 and not on MI300, apologies. Two tests in https://github.com/vllm-project/vllm/pull/49747 were wrongfully implemented when running on gfx942: - test_gfx950_picks_ai…

🦀 Rust Frontend

  • e04a30a #49914 — [Frontend] Lazily initialize chat media connectors (#49914)
    • 作者: Andreas Karatzas | +43/-11 | 2 个文件
    • Defer media connector construction until a request contains multimodal input. - Avoid repeated shared-cache initialization for text-only chat parsing while covering synchronous and asynchronous paths.
  • 0a31372 #47301 — [Frontend] Add detokenization streaming derender for disaggregated serving (#47301)
    • 作者: Martin Hickey | +1377/-32 | 6 个文件

    Part 1 of 3 for RFC #47161. Adds a streaming mode to the derender/ endpoints which mirrors the producer’s per chunk SSE shape. Streaming is a requirement for interactive serving on platforms like llm-d and Dynamo. Points to note: - State is kept client carried and stateless on the server - Responses are returned as plain JSON rather than SSE. This is because the derender can’t buffer a whole gener…

  • 9d1aa4d #48543 — [Frontend] Add diarized_json support for MOSS-Transcribe-Diarize (#48543)
    • 作者: christian | +309/-10 | 9 个文件

    closes #48443. Add OpenAI-compatible diarized_json responses to /v1/audio/transcriptions for models that explicitly support diarized transcription. OpenMOSS-Team/MOSS-Transcribe-Diarize is the first supported model. MOSS emits a compact transcript in the form [start][Sxx]text[end]. This change parses that model-specific format after generation and returns OpenAI-style speaker-attributed segments w…

  • 6370e53 #48145 — [Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145)
    • 作者: Seiji Eicher | +198/-13 | 6 个文件

    Why In prefill and decode disaggregation, the prefill stage renders the prompt from messages and tokenizes it. The router forwards the same chat request to the decode stage, which renders and tokenizes it a second time. For long prompts, that repeated render and tokenize adds latency on the decode critical path. The decode stage does not need to redo it. The prefill response already contains th…

📖 Documentation

🐛 Bug Fix

  • 4e582c5 #50326 — [PD][Bugfix] Rebase KV lease deadlines onto worker clock (#50326)
    • 作者: Nick Hill | +76/-0 | 5 个文件

    pull_scheduler.request_finished stamps reqs_to_send deadlines with the scheduler process’s time.perf_counter(); base_worker expires them against each worker’s own perf_counter. perf_counter epochs are process-local — across nodes they differ by boot-time deltas, so on a multi-node prefill deployment the second node’s effective TTL is ttl − (node clock-epoch gap), which can be near zero or …

  • 437e0b7 #50297 — [BugFix] Fix P/D preemption race condition (#50297)
    • 作者: Nick Hill | +117/-5 | 7 个文件

    There is a small race condition with P/D and async scheduling, where a prefill request can be preempted during it’s final (possible only) step. The scheduler logic currently allows such requests to complete normally because for most purposes they are finished (final output token generated and returned). They just get removed from the waiting queue. However at this point, their kv blocks have been …

  • 451227c #49660 — [Bugfix][Kernel] Fix integer overflow in libtorch_stable/activation_kernels.cu (#49660)
    • 作者: Molly | +9/-6 | 1 个文件

    Fixes #42860. Several kernels in csrc/libtorch_stable/activation_kernels.cu calculate pointer offsets using expressions such as: Because blockIdx.x and d participate in 32-bit arithmetic, these expressions can wrap when processing sufficiently large numbers of tokens and hidden dimensions. The resulting incorrect pointer offsets may cause the kernels to access unintended memory. This PR converts b…

  • 48aa8d8 #41357 — [Bugfix] Prevent stale multiproc RPC deadlines from becoming unbounded waits (#41357)
    • 作者: bugkeep | +175/-2 | 3 个文件

    Prevents stale MultiprocExecutor.collective_rpc() deadlines from becoming negative response-queue timeouts that can be misinterpreted as unbounded waits. Supersedes #41213 (minimal inline clamp only) with a standalone _remaining_timeout() helper, sync-path bypass of FutureWrapper, a recv() lower-level guard, and 25 comprehensive tests. ## Root cause collective_rpc() computes an absolute deadline a…

  • a0c092e #48245 — [BugFix] Fix num_output_placeholders preemption underflow (#48245)
    • 作者: Nick Hill | +385/-77 | 5 个文件

    Fix an EngineCore crash under async scheduling + KV-pressure preemption (most visible with spec decode / PP at high concurrency): This reworks #47900, which found the same bug and put the fix in the right place (_preempt_request) but chose to discard the request’s in-flight output — an approach that was incomplete and not lossless (its CI failed: test_preempt_defers_free_and_clears_boo…

  • e0cfa52 #49073 — [Bugfix][Frontend] Return transcription and translation verbose as float (#49073)
    • 作者: christian | +4/-4 | 3 个文件

    Fixes #49068. Return duration as a number in verbose_json transcription and translation responses, matching the OpenAI-compatible response contract and allowing typed clients to deserialize it correctly. Manual endpoint validation with openai/whisper-large-v3-turbo: ## Test Result Both endpoints returned duration as a JSON number: /v1/audio/transcriptions: /v1/audio/translations: —

  • aeaa50a #49975 — [Bugfix][Multimodal] Include media IO config in MM cache hash (#49975)
    • 作者: Guan-Ming Chiu | +54/-3 | 5 个文件
    • Image hash ignores decode config, so same bytes + different image_mode / rgba_background_color collide, returning another request’s cached features - Fix: attach io_config to MediaWithBytes only when decoding altered the image; default hashes unchanged - Also route the no-mm_uuids branch through get_all_items_for_hash() for consistency - Orthogonal to #49686; happy to rebase if it lands first - …
  • 32e657e #49343 — [BugFix] eagle draft max position embeddings (#49343)
    • 作者: Jared Wen | +288/-4 | 4 个文件

    #48894 cc @manojarulmurugan @Isotr0py —

🔩 Misc

  • 2ecd864 #50092 — Revert “[Misc][Minimax-M3]add default video_processor (#50092)” (#50313)
    • 作者: Nick Hill | +7/-44 | 2 个文件

    Add MiniMaxM3VLVideoProcessor to minimax-m3 Automatically select registered video loader for VLM. ## Test Result —

  • df2735e #50092 — [Misc][Minimax-M3]add default video_processor (#50092)
    • 作者: rongfu.leng | +44/-7 | 2 个文件

    Add MiniMaxM3VLVideoProcessor to minimax-m3 Automatically select registered video loader for VLM. ## Test Result —

⚡ Performance

  • 82642d7 #49750 — [Perf] RMSNorm uncontiguous support, 1.2~3.1x kernel performance improvement (#49750)
    • 作者: Wentao Ye | +31/-29 | 9 个文件

    Thanks @tlrmchlsmth for the context Part of https://github.com/vllm-project/vllm/issues/46654, mainly optimization for GLM5.2, but found that similar issues appear in a lot of other models so fixed accordingly. Covered in current unit tests Perf can be seen in this AI generated script And we will get

  • d6247d7 #49731 — [Spec Decode][Perf] Replicate DSpark Markov head across TP ranks (#49731)
    • 作者: Michael Goin | +94/-23 | 4 个文件

    Replicate the DSpark Markov embedding and projection across TP ranks, removing an all-reduce and full-vocabulary gather per draft position. ## Benchmark Qwen3-14B DSpark on B300, batch size 1, 32 input / 128 output tokens, 7 speculative tokens, dummy weights, 3 warmups and 10 measured iterations: | TP | Sharded | Replicated | Gain | |—:|—:|—:|—:| | 2 | 161.5 tok/s | 166.8 tok/s | +3.3% | |…

🖥️ Kernel

  • f51193b #49291 — [Kernel][Mamba] Fused-kernel support for align-mode DS-conv state migration with num_accepted_tokens > 1 (#49291)
    • 作者: sungsoo ha | +409/-64 | 5 个文件

    Hybrid/Mamba models support an align-mode Mamba cache path where the running Mamba state is migrated at scheduler-step / block boundaries. For DS conv-state layout, the align path has an intentionally guarded, not-yet-implemented cell: accepted-token tail copies with num_accepted_tokens > 1 (MTP degree > 1). The scalar get_conv_copy_spec helper rejects this case: Root cause. DS conv-state …