共 80 个 commit，涉及 535 个文件，+47162/-3631 行变动。

概要

统计项	数值
Commit 数	80
变更文件	535
新增行数	+47162
删除行数	-3631

Commit 列表

🧪 CI/Tests

• 381b691 #50262 — [ROCm][CI] Fix Kimi K3 KDA on ROCm (#50262)
  • 作者: stefankoncarevic | +8/-2 | 2 个文件

  The AMD “Kernels KDA Test” job (mi300_1) was red. Investigation uncovered two separate problems, fixed here in two commits: 1. Stale CI path the job pointed at a test file that was moved by another PR, so it failed before running anything. 2. ROCm false-positive kernel selection once the job was repointed and actually ran, it surfaced a genuine ROCm bug: FlashKDA (a CUDA-only extension) wa…

• 625871b #50241 — [CI][Test] Fix pooling truncation test after VLLMError hierarchy change (#50241)
  • 作者: stefankoncarevic | +3/-1 | 1 个文件

  tests/models/language/pooling/test_truncation_control.py::test_bigger_truncation_size began failing on main. The test asserts pytest.raises(ValueError) when truncate_prompt_tokens > max_model_len, but the request-validation path now raises VLLMValidationError, which is no longer a subclass of ValueError, so the exception is not caught and the test fails. This updates the test to expect VLLMValidat…

• 5b14019 #50222 — [CI] Fix MXFP8 MOE backend selection tests on gfx942 (#50222)
  • 作者: fxmarty-amd | +11/-7 | 1 个文件

  As reported by @jikunshang (https://github.com/vllm-project/vllm/pull/49747#issuecomment-5114449319), #49747 includes two new failing tests, fixing in this PR. These failures were overlooked as local testing of #49747 was done on MI350 and not on MI300, apologies. Two tests in https://github.com/vllm-project/vllm/pull/49747 were wrongfully implemented when running on gfx942: - test_gfx950_picks_ai…

• 100d655 #50211 — [CI] Allow PR comment acknowledgements (#50211)
  • 作者: Kevin H. Luu | +1/-1 | 1 个文件

  Grant the comment-trigger workflow explicit pull-requests: write permission so it can acknowledge commands and add reactions on pull request conversations. ## Root cause The live /ci run test successfully created and completed Buildkite build #80923, but the workflow failed when it tried to add the acknowledgement comment: The eyes, rocket, and fallback confused reactions failed with the same resp…

• 7de49ba #48703 — [XPU][UT][CI] add xpu config to run gpt-oss accuracy in ut and ci (#48703)
  • 作者: zofia | +38/-0 | 4 个文件
• 6f91edf #49974 — [Test] dynamic_shapes_compilation (#49974)
  • 作者: Jared Wen | +46/-56 | 1 个文件

  replace LLM() with VllmRunner() for a porper shutdown along the testing —

• 0bb548b #50161 — [CI][ROCm] Stabilize Qwen2-VL LoRA test (#50161)
  • 作者: Andreas Karatzas | +21/-2 | 2 个文件

  Stabilize and re-enable LoRA %N gating mirror. Motivation: - https://buildkite.com/vllm/ci/builds/80664/canvas?jid=019fa6d8-48e1-4083-8f46-b95c89e802a4&tab=output

• 7398a30 #50190 — [ROCm][CI] Stabilize ngram and suffix correctness test (#50190)
  • 作者: Andreas Karatzas | +19/-8 | 1 个文件

  Stabilizes the ROCm test_ngram_and_suffix_correctness group after the intermittent AMD CI failure in Buildkite build 11367. The failing run completed the ngram case, force-killed its engine after the five-second shutdown window, and then failed in the suffix case with an asynchronous GPU fault. The retry passed, so this PR does not claim a confirmed production-kernel root cause. It isolates persis…

• 54ab69b #50197 — [CI] Allow comment-triggered builds past pipeline filters (#50197)
  • 作者: Kevin H. Luu | +2/-0 | 2 个文件

  • Set ignore_pipeline_branch_filters on Buildkite builds created by /ci run. - Extend the build-payload contract test to require the override. ## Root cause The live /ci run test reached Buildkite after the API token was corrected, but Buildkite rejected build creation with: The pipeline provider filter only accepts main or PRs labeled ready/ready-run-all-tests. That conflicts with the intended re…

• 7f4c52f #50132 — [CI] Add comment-based Buildkite triggers (#50132)
  • 作者: Kevin H. Luu | +989/-5 | 6 个文件

  Replace label-driven full CI triggering with explicit PR comment commands so reviewers and authors can request CI only when needed. - /ci run creates a Buildkite build for the current PR head SHA. - /ci retry retries only failed, timed-out, or expired jobs from the latest build for the current PR head. - Users with write, maintain, or admin permission, plus users listed in the optional CI_TRUSTED_…

• fe65aa6 #50171 — [CI][NIXL] Fix flaky DP+EP test port conflict (#50171)
  • 作者: Divakar Verma | +4/-1 | 1 个文件

  This PR fixes flaky NIXL accuracy test timeout in DP+EP mode by setting VLLM_DP_MASTER_PORT for the decode instance. Without this PR, the DP_EP case for decode instance has tendency to fall into race-condition as the ports are not explicitly set for this case. They might end up inheriting the same fixed base port and might compete with the prefill instance’s port range. Fix: Set VLLM_DP_MASTER_POR…

• 176256b #50163 — [ROCm][CI] Stabilize ROCm audio streaming test (#50163)
  • 作者: Andreas Karatzas | +6/-0 | 1 个文件

  Stabilize Entrypoints Integration (Multimodal). Motivation: - https://buildkite.com/vllm/amd-ci/builds/11367/list?jid=019fa7f3-f1d2-41e0-94a2-936940f42672&tab=output

• e7f6a39 #50110 — [Test] Make EPD correctness tests configurable for XPU (#50110)
  • 作者: liuzhenwei | +64/-31 | 2 个文件

  • Add XPU device affinity support to the EPD correctness test. - Make gpu-memory-utilization max-num-seqs max-model-len configurable. USE_TWO_IMAGE_PROMPT=0 DEVICE_PLATFORM=xpu ZE_AFFINITY_MASK=0,1,2,3 GPU_MEMORY_UTILIZATION=0.8 bash run_epd_correctness_test.sh ## Test Result —

• 94100b5 #49340 — [CI] Wire untethered test files into CI jobs (#49340)
  • 作者: Nick Hill | +88/-13 | 9 个文件

  An audit of .buildkite pytest targets against tests/**/test_*.py found dozens of test files that no CI job ever runs. Wire in the ones that pass on current main (every set validated on B200-class hardware, or CPU-only for CPU tests, before wiring): - tests/kernels root: after the kernels/ subdir categorization (#16799), the root stopped being collected, and 19 files added since then never ran. Add…

• 25ace8f #49881 — [CI] Increase Qwen3.5 MTP GSM8K generation length (#49881)
  • 作者: Jiangyun Zhu | +2/-1 | 1 个文件

  The Qwen3.5 MTP3 GSM8K evaluation is frequently truncated by the default max_tokens=256. The evaluator extracts the last integer from the generated text even when the reasoning is incomplete, so a truncated chain of thought can be scored as a valid but incorrect answer. This caused the MTP configuration to fail in Buildkite build #80270: - Qwen3.5-397B-A17B-NVFP4-DEP2-MTP: 0.8469 accuracy, 5.8% in…

• 247470f #48164 — [CI] Add PyTorch stable ABI audit check (#48164)
  • 作者: Chris Leonard | +130/-1 | 6 个文件

  • Add a new Buildkite step, Torch Stable ABI Audit, that runs after the CUDA CI image build and uses torch-abi-audit to verify vLLM’s compiled extensions comply with the PyTorch stable ABI. - Fail CI if any extension links unstable libtorch symbols (at:: / c10:: / etc.) unless it is listed in ALLOWED_UNSTABLE_LIBRARIES in .buildkite/check-torch-abi.py. - Fail CI if an allowlist entry is stale …

• 88402a4 #49945 — [Test] Skip ROCm AITER MLA prefill tests on non-ROCm platforms (#49945)
  • 作者: Liangliang Ma | +6/-0 | 1 个文件

  TestROCmAiterFAPrefillSelection exercises ROCm-specific AITER FlashAttention MLA prefill backend gating. On non-ROCm platforms (e.g. XPU) patching vllm.platforms.rocm forces its module-level import, which runs torch.cuda.get_device_properties and fails when torch is not built with CUDA. Guard the class with a ROCm-only skipif.

• 99b57a4 #50086 — [CI][ROCm] Soft fail LoRA mirror (#50086)
  • 作者: Andreas Karatzas | +1/-0 | 1 个文件

  Soft fail LoRA mirror until proper fix on ROCm.

⚡ Performance

• d6247d7 #49731 — [Spec Decode][Perf] Replicate DSpark Markov head across TP ranks (#49731)
  • 作者: Michael Goin | +94/-23 | 4 个文件

  Replicate the DSpark Markov embedding and projection across TP ranks, removing an all-reduce and full-vocabulary gather per draft position. ## Benchmark Qwen3-14B DSpark on B300, batch size 1, 32 input / 128 output tokens, 7 speculative tokens, dummy weights, 3 warmups and 10 measured iterations: | TP | Sharded | Replicated | Gain | |—:|—:|—:|—:| | 2 | 161.5 tok/s | 166.8 tok/s | +3.3% | |…

• a07fac7 #48442 — [Perf] Zero-copy torch.Tensor pickling in shm_broadcast MessageQueue (#48442)
  • 作者: Ruinan Ma | +233/-3 | 3 个文件

  torch.Tensor.reduce_ex copies tensor bytes into the pickle stream and never emits a PickleBuffer, so the out-of-band buffer path added to MessageQueue.enqueue in #26961 is dead code for tensors. Large CPU tensors broadcast from the engine core to workers — e.g. prompt_embeds in SchedulerOutput with the multiproc executor — are byte-copied once by the engine on the scheduling critical path, and…

🐛 Bug Fix

• a0c092e #48245 — [BugFix] Fix num_output_placeholders preemption underflow (#48245)
  • 作者: Nick Hill | +385/-77 | 5 个文件

  Fix an EngineCore crash under async scheduling + KV-pressure preemption (most visible with spec decode / PP at high concurrency): This reworks #47900, which found the same bug and put the fix in the right place (_preempt_request) but chose to discard the request’s in-flight output — an approach that was incomplete and not lossless (its CI failed: test_preempt_defers_free_and_clears_boo…

• e0cfa52 #49073 — [Bugfix][Frontend] Return transcription and translation verbose as float (#49073)
  • 作者: christian | +4/-4 | 3 个文件

  Fixes #49068. Return duration as a number in verbose_json transcription and translation responses, matching the OpenAI-compatible response contract and allowing typed clients to deserialize it correctly. Manual endpoint validation with openai/whisper-large-v3-turbo: ## Test Result Both endpoints returned duration as a JSON number: /v1/audio/transcriptions: /v1/audio/translations: —

• aeaa50a #49975 — [Bugfix][Multimodal] Include media IO config in MM cache hash (#49975)
  • 作者: Guan-Ming Chiu | +54/-3 | 5 个文件

  • Image hash ignores decode config, so same bytes + different image_mode / rgba_background_color collide, returning another request’s cached features - Fix: attach io_config to MediaWithBytes only when decoding altered the image; default hashes unchanged - Also route the no-mm_uuids branch through get_all_items_for_hash() for consistency - Orthogonal to #49686; happy to rebase if it lands first - …

• 32e657e #49343 — [BugFix] eagle draft max position embeddings (#49343)
  • 作者: Jared Wen | +288/-4 | 4 个文件

  #48894 cc @manojarulmurugan @Isotr0py —

• 6fbbcf2 #49757 — [BugFix] Stop dummy runs from writing mamba state through stale block-table rows (#49757)
  • 作者: Nick Hill | +86/-2 | 5 个文件

  Dummy-run batches execute real state-writing kernels, and both model runners mask only the attention side (PAD-filled slot mappings / null routing) while the mamba/GDN/KDA path routes its in-place state writes through block_table[:, 0]. Two paths let stale block ids reach those writes: - V1: condense() vacates the moved-from row without clearing it, leaving a duplicate of a live request’s block id…

• 56f31af #41602 — [Bugfix] Fix /wake_up crash on hybrid models (Mamba/DeltaNet) (#41602)
  • 作者: Kevin Glynn | +66/-3 | 2 个文件

  Fix AttributeError: ’list’ object has no attribute ‘zero_’ when calling /wake_up after /sleep on hybrid models that use Mamba, DeltaNet, or other MambaSpec-based layers with quantized KV cache. Fixes #41564 ## Root Cause init_fp8_kv_scales() (called via post_kv_cache_wake_up() → wake_up()) iterates self.kv_caches and calls .zero_() on each entry, assuming every entry is a torch.Tensor: But for hyb…

• 0b6aa3c #50065 — [Bugfix][Spec Decode] Size DFlash query buffers for cudagraph-padded batches (#50065)
  • 作者: Siddhant Bharti | +7/-3 | 1 个文件

  Fix a startup crash (and a latent runtime overflow) in DFlash speculative decoding on the V1 GPU model runner for any odd –max-num-seqs. DFlashProposer.init allocates its persistent query buffers (positions, _slot_mapping_buffer) to exactly max_query_tokens = max_num_seqs * (1 + num_speculative_tokens) (vllm/v1/spec_decode/dflash.py). But _determine_batch_execution_and_padding → cudag…

• 118bcde #45532 — [BugFix] Fix clang spinloop mwaitx include (#45532)
  • 作者: johnnyychiu | +1/-1 | 1 个文件

  Fixes #45515. csrc/spinloop.cpp included <mwaitxintrin.h> directly for the x86 MONITORX/MWAITX intrinsics. Clang rejects that header when included directly and asks callers to include <x86intrin.h> instead. Switching to the umbrella x86 intrinsic header keeps _mm_monitorx and _mm_mwaitx available while allowing clang-hosted builds to compile the spinloop target. - Inspect the resulting diff to con…

• 6c7e679 #49714 — [ROCm][Bugfix] Sanitize AITER paged-MQA logits before sparse top-k for DeepSeek-V4 (#49714)
  • 作者: Shanshan Shen | +65/-0 | 2 个文件

  Fix: https://github.com/Fangzhou-Ai/vllm/issues/13 ### Summary This change prevents a GPU memory access fault when serving DeepSeek V4 with ROCm sparse attention on gfx950 under concurrent mixed prefill/decode workloads. The ROCm AITER deepgemm_fp8_paged_mqa_logits kernel can intermittently return NaN or infinite values within the valid logits range. The generic histogram top-k kernel assumes fini…

• 1db989b #49030 — [Bugfix][Multimodal] Fix video temporal padding estimates (#49030)
  • 作者: labAxiaoming | +34/-9 | 7 个文件

  Fix Video Temporal Padding Token Estimates Fix incorrect video token estimation when temporal_patch_size > 2. Fix #47866 ## Root Cause Several multimodal models used: This only works reliably for temporal_patch_size 1 or 2. For example: The incorrect padding can make grid_t and the estimated video token count too small. This affects vLLM-side budget estimation, dummy input sizing, and warmup. Re…

• 6453fc0 #50053 — [Bugfix] Don’t reuse engine core payload buffer while zmq is sending it (#50053)
  • 作者: Nick Hill | +179/-58 | 4 个文件

  EngineCoreProc.process_output_sockets recycles the msgpack payload bytearray across messages via MsgpackEncoder.encode_into, gated on the tracker returned by send_multipart(copy=False, track=True). That gate never engages. Socket.send_multipart() returns a tracker for the last frame only, and pyzmq copies any frame smaller than zmq.COPY_THRESHOLD (64KiB), returning the always-done _FINISHED_TRAC…

• 30217b0 #49877 — [Bugfix][KV Offload][P2P] Scope serve state to fetch rounds (#49877)
  • 作者: Itay Etelis | +573/-393 | 6 个文件

  FIX #49820 The server keeps a single outbound slot per kv_request_id, but a request can go through several lookup/fetch rounds as its prefix grows. When rounds overlap, finishing one round wipes the supply that was already pinned for the next one, so the next fetch either stalls for the full 30s load timeout (the trace in the issue) or trips the duplicate-fetch guard and takes down the whole sessi…

• 1e81853 #49964 — [Bugfix][KV Offload] Keep Mamba block span unscaled under DCP (#49964)
  • 作者: Jonguk Cheong | +66/-2 | 2 个文件

  Native KV offloading currently multiplies every cache group’s token span by the decode context parallelism world size. This is correct for attention KV, which is sharded across DCP ranks, but Mamba state is replicated and keeps its original block size. With 16 token attention and Mamba blocks under DCP=2, the current configuration derives 32 tokens per block for both groups. The correct spans are …

• b6cbba8 #48391 — [Bugfix][Kernel] Fix batch invariance in RMSNorm kernels by pinning block size (#48391)
  • 作者: oops-oom | +189/-21 | 6 个文件

  Fix a batch-invariance gap in the fused_add_rms_norm CUDA kernel. The kernel selects its block size from the token count: max_block_size = (num_tokens < 256) ? 1024 : 256, independent of whether batch-invariant mode is enabled. Under VLLM_BATCH_INVARIANT=1, the residual RMSNorm path (RMSNorm.forward_cuda with a residual → ops.fused_add_rms_norm) routes to this kernel. As a result the same token …

• 62d8db7 #50131 — [Bugfix] Add missing vllm/models/kimi_k3/__init__.py (#50131)
  • 作者: Harry Mellor | +4/-0 | 2 个文件

  #50090 added vllm/models/kimi_k3/ but never added vllm/models/kimi_k3/init.py, so the directory is an implicit namespace package. This broke the docs build on main: Failing build: https://app.readthedocs.org/projects/vllm/builds/33792546/ (at 61ac368). api-autonav walks the source tree and finds the subpackages, but Griffe cannot resolve vllm.models.kimi_k3.amd because its parent has no __in…

• 948107a #48589 — [Bugfix] Enhance extra_config handling for layer name suffix matching (#48589)
  • 作者: Xin He | +57/-0 | 2 个文件

  This pull request improves how the configuration is selected for layers with extra configuration by adding support for suffix matching. This addresses situations where the keys in extra_config are short names (like “lm_head”), but the actual layer_name is fully qualified (like “model.language_model.lm_head”), which can happen due to model nesting. Configuration handling improvements: * Added s…

• b09688a #49774 — [Bugfix][Spec Decode] Preserve draft buffers across level-2 sleep (#49774)
  • 作者: aoshen02 | +12/-11 | 1 个文件

  Fixes two problems in the level-2 sleep/wake buffer-restore path, and removes a mechanism that the first fix makes redundant. 1. Speculative draft-model buffers were never snapshotted. Level-2 sleep releases the weights pool outright (offload_tags=()), so on resume the pages are remapped but uninitialized. Registered buffers are not recoverable from the checkpoint (cos_sin_cache and friends ar…

📦 Other

• 43eaefb #48791 — [ModelRunner V2] Enable sequence pooling for embedding and classification models (#48791)
  • 作者: Taneem Ibrahim | +509/-37 | 10 个文件

  Fixes the sequence-level portion of https://github.com/vllm-project/vllm/issues/41286 and unblocks that portion of https://github.com/vllm-project/vllm/pull/48290. After https://github.com/vllm-project/vllm/pull/49331 added the encoder-only attention and model-state foundation, Model Runner V2 still did not execute a model’s pooler: it hardcoded last-token normalization, assumed every pooling mode…

• 242c591 #49341 — [Rust Frontend] Send multimodal tensors in auxiliary frames (#49341)
  • 作者: Reid | +530/-159 | 11 个文件

  Rust frontend EngineCoreRequest::Add messages currently convert multimodal tensors into owned raw bytes, copy those bytes into one large msgpack payload, and copy that payload again into the zeromq send buffer. Large images, multiple images, and video requests can therefore require several full-size memory copies and one large contiguous msgpack allocation. Although the wire representation already…

• 72297d8 #47121 — [XPU] Route weightless RMSNorm to _C dispatch (#47121)
  • 作者: Yintong Lu | +0/-6 | 1 个文件

  Background After vllm-xpu-kernels adds support for weightless RMSNorm, vLLM dispatch should no longer send XPU weight=None calls to native fallback paths. Currently, if dispatch still falls back (for weight=None), the new XPU kernel capability is not fully utilized. ### Problem - Kernel capability exists in vllm-xpu-kernels. - vLLM dispatch path may still use native/IR fallback for XPU weight=…

• 542a8fa #50094 — [KV Offload] Move CPUOffloadingSpec onto SharedOffloadRegion (#50094)
  • 作者: Chang Guo | +132/-5 | 2 个文件

  Move the default KV-offload backend CPUOffloadingSpec’s worker-side CPU buffer from a per-rank private pinned torch tensor onto the existing shared SharedOffloadRegion mmap, on CUDA/ROCm. This is the allocation-swap prerequisite for the TP-deduplication feature requested in #47929; it introduces no deduplication semantics on its own. Maintainer authorization (upstream reviewers do not see the …

• 9a4e5f9 #43538 — [CI/Perf] Fix malformed serving benchmark config (#43538)
  • 作者: Minh Vu | +0/-5 | 1 个文件

  Fixes #43537. This removes a stray duplicate object from .buildkite/performance-benchmarks/tests/serving-tests.json that makes the default serving benchmark config invalid JSON. ## Root cause PR #43262 converted the config to the newer defaults-based format but left an extra dataset_name / dataset_path object after serving_llama8B_tp1_sharegpt. ## Validation Ran locally: The JSON parses successful…

• c44e191 #49604 — [Rust Frontend] Add –limit-mm-per-prompt support (#49604)
  • 作者: Maria Guevara | +413/-16 | 14 个文件

  Adds –limit-mm-per-prompt as an optional CLI flag, capping how many input items of each modality (image, image_embeds, audio, video) appear per chat request. Modalities absent from the map default to unlimited. Rejects with HTTP 400 in validate_mm_limits, before fetch_media runs. This matches the python behavior, which validates item counts per modality against limit_per_prompt in vllm/config/mul…

• dad7a63 #49582 — [EC Connector] Add has_pending_push_work (#49582)
  • 作者: omerpaz95 | +16/-0 | 2 个文件

  Adds has_pending_push_work() to ECConnectorBase, mirroring the existing KV connector API (vllm/distributed/kv_transfer/kv_connector/v1/base.py). This gives EC connectors a hook to signal that push-mode transfer work is still in flight, and wires it into Scheduler.has_requests() so the engine main loop keeps stepping until any pending EC push work drains. Default implementation returns False, so co…

• 5b29c95 #48677 — [XPU] upgrade to torch 2.13 (#48677)
  • 作者: Yan Ma | +35/-92 | 7 个文件

  This PR upgrades to torch 2.13 on XPU platform and bump up v0.1.12 vllm-xpu-kernels. Note that dev environment is not installed. ## Test Result —

• ad5d29d #50210 — [Model] Support Qwen3.5 text-only dense and MoE models (#50210)
  • 作者: Perkz Zheng | +62/-0 | 5 个文件

  • Register Qwen3_5ForCausalLM and Qwen3_5MoeForCausalLM as text-generation architectures. - Register the qwen3_5_text and qwen3_5_moe_text Hugging Face configuration types. - Mark the causal-LM base as hybrid and provide the GDN state dtype, shape, and copy hooks required by hybrid scheduling. - Reuse the existing Qwen3.5 cache-dtype handling and remove multimodal M-RoPE fields for text-only causa…

• f5a7cce #50093 — [Model] Add Kimi K3 support: Python frontend [2/2] (#50093)
  • 作者: Bugen Zhao | +3056/-5 | 20 个文件

  Extract the Kimi K3 Python frontend from #50000 as the top layer of the independent frontend stack rooted at main through #50104. This layer adds Python XTML rendering, reasoning and tool parsing, structural-tag handling, and streaming support. Model-side preprocess_messages cleanup lives in #50089. This PR contains no vllm/models/kimi_k3 or rust/ changes. An open-PR search found this as the sole …

• 65a1a16 #50194 — [CPU] Fix FP8 attention scratchpad sizing (#50194)
  • 作者: Tianmu Li | +35/-7 | 5 个文件

  The CPU attention scheduler computed FP8 KV tile geometry using sizeof(kv_cache_t). With BF16 queries and one-byte FP8 KV cache entries, this selected larger tiles than the BF16-backed scratchpad can hold during large AMX prefills. The resulting out-of-bounds writes cause corrupted output Unit Model ## Test Result | Metric | Before | After | |—|—:|—:| | Strict / flexible | 0.67 / 0.67 | 0.87…

• 7c6729b #50089 — [Model] Add Kimi K3 support: model files and kernels [1/N] (#50089)
  • 作者: Jiangyun Zhu | +28009/-773 | 109 个文件

  split https://github.com/vllm-project/vllm/pull/50000 Note: it’s not runnable now ## Test Result —

• 6f00a1a #42436 — fused_moe: add VLLM_TRITON_USE_TD tensor-descriptor path (#42436)
  • 作者: Artur Fierka | +179/-21 | 4 个文件

  Working on perf optimizations for the Triton MoE kernels. Adds an opt-in VLLM_TRITON_USE_TD env var that switches the fused MoE kernel onto a tensor-descriptor based load/store path, mirroring VLLM_TRITON_ATTN_USE_TD (PR #40327). Auto-on for XPU; off by default on CUDA/ROCm (opt-in on Blackwell). The changes in this PR are scoped to the fused MoE kernel (fused_moe_kernel) only. The env var name fo…

• db7a79c #50144 — [CPU] Fix s390x builds and update torch version in dockerfile (#50144)
  • 作者: Rehan Khan | +5/-5 | 2 个文件

  Fix s390x builds and upgrade the torch version to 2.13.0 in dockerfile Build the server and send request ## Test Result —

• dc1be79 #49114 — Add CachePolicyFactory for pluggable/external eviction policies (#49114)
  • 作者: Philip Pesic | +269/-33 | 15 个文件

  CPUOffloadingManager resolves eviction_policy (“lru”/“arc”) through a private, closed dict in manager.py with no way for an out-of-tree package to plug in its own CachePolicy implementation by name, the same way OffloadingSpecFactory and KVConnectorFactory already let external specs/connectors register themselves. Without this, the only way to use a custom CachePolicy is to construct CPUOffloading…

• f37f03d #49762 — [KV Connector] Support NIXL P/D for hybrid MLA+SSM models (#49762)
  • 作者: Nick Hill | +281/-38 | 5 个文件

  Re-implements the intent of https://github.com/vllm-project/vllm/pull/44848 on top of the reworked NIXL connector, from the current architecture rather than the original patch. KimiLinear pools its KDA (GDN-typed MambaSpec) and MLA layers into shared HMA tensors, making every region dual-purpose. Since the mamba-page unification raises the attention block size until the MLA page equals the unified…

• 32a423a #49580 — Integrate CuTeDSL MoE for ReLU2 NVFP4 (#49580)
  • 作者: danielafrimi | +245/-9 | 3 个文件

  GSM8K passed at 0.9431 vs 0.9300 for Super NVFP4 model. Related reference: - SGLang FlashInfer one-sided A2A + CuTeDSL MoE integration for Nemotron Ultra: https://github.com/sgl-project/sglang/pull/28309 ## A2A serving benchmark Ran an A2A comparison for nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with CUDA graphs enabled, FlashInfer autotune enabled. Workload/config: - GPUs: 4 total, –data-pa…

• 30c2718 #43229 — [CompressedTensors] FP4 Qutlass Integration (#43229)
  • 作者: Kyle Sayers | +169/-34 | 7 个文件

  Summary ## This PR integrates the NVFP4 Qutlass kernel for compressed-tensors models with NVFP4 weights and online transforms. It supports any block-diagonal transform, and is not limited to deterministic hadamard transforms like hadacore. It is selected as the default, over hadacore, for any applicable configuration. To reduce memory usage, online-transform weights are shared where applicable….

• 17a74b7 #48876 — [Model] Add Inkling compressed-tensors dynamic FP8 support (#48876)
  • 作者: Krishna Teja Chitty-Venkata | +29/-0 | 2 个文件

  Add the remaining Inkling weight-loader support needed by RedHatAI/Inkling-FP8-dynamic on top of the Inkling compressed-tensors infrastructure now available on main. The dynamic FP8 checkpoint stores w2_weight_scale as per-output-channel [experts, output_channels, 1] scales. Inkling tensor parallelism shards the w2 input dimension, so these scales must be replicated on every TP rank. Previously th…

• 5369f7b #49747 — [MXFP8][ROCm] Fix MXFP8 MoE backend selection (#49747)
  • 作者: fxmarty-amd | +244/-61 | 6 个文件

  TL;DR This PR: - Fixes wrongful dispatch on AITER_MXFP8 MOE backend, for unsupported activation/alpha/beta. - Fixes wrongful hard-coded alpha/beta in TRITON_MXFP8 and EMULATION MOE mxfp8 backends. - Fixes oracle/mxfp8.py dispatch logic to remove ROCm-specific logic that is already handled by is_supported_config. This PR fixes a longstanding failure in AMD MI355 CI: e.g. in https://buildkite.com…

• bb3b61f #49618 — perf: dispatch non-grouped bias-less topk routing methods to fused path (#49618)
  • 作者: Julien Debache | +159/-23 | 2 个文件

  create_fused_moe_router routed every use_grouped_topk=True model to GroupedTopKRouter, including configs with one expert group and topk_group=1. With a single group the masking selects all experts, so the grouped implementation computes softmax, group-max, top-1-over-one-group, zeros_like, scatter_, expand and masked_fill to produce what a single fused top-k kernel produces directly. The fused gro…

• 2899dca #50104 — [Model] Add Kimi K3 support: Rust frontend [1/2] (#50104)
  • 作者: Bugen Zhao | +3003/-39 | 35 个文件

  Extract the Kimi K3 Rust frontend from #50000 into an independently reviewable stack rooted at main. This layer adds native XTML rendering, unified parsing, tokenizer/model selection, structured-tag support, and top-level response_format forwarding. Model and kernel work remains in #50089. This PR contains only rust/ changes. An open-PR search found this as the sole Kimi K3 Rust frontend PR, so it…

• d552a68 #50129 — [Rust Frontend] Extract shared tracing setup logic into vllm-tracing (#50129)
  • 作者: Bugen Zhao | +35/-21 | 8 个文件

  #48937 introduced tracing-based benchmark logs and left separate subscriber setup in vllm-cmd and standalone vllm-bench. This PR adds a vllm-tracing workspace crate containing the existing vLLM-style formatter, environment filter, and init_tracing(process_label) entrypoint. Both binaries now use the shared implementation: - vllm-rs bench and standalone vllm-bench emit the same vLLM-style log forma…

• 8a7b3c2 #49483 — [compressed-tensors] update find_matched_target order to prioritize fused name matches over class match (#49483)
  • 作者: Brian Dellabetta | +1/-1 | 1 个文件

  Resolves https://github.com/vllm-project/llm-compressor/issues/2730 Currently on main, the order of target resolution is 1. match name 2. match class 3. match fused name This PR flips 2 and 3, matching the logic of compressed-tensors. This is helpful when users have mixed-precision quantization configs and aren’t including all fused names in their targets, and have a last config group targeting Li…

• 05a0814 #49906 — [ROCm] Fix and optimize GPT-J-style MRoPE (#49906)
  • 作者: Andreas Karatzas | +100/-59 | 2 个文件

  • Honor the existing adjacent-pair MRoPE setting instead of hardcoding NeoX pairing. - Use contiguous ROCm loads and stores with register-only split/interleave, preserving the NeoX and non-ROCm launch paths.

• 4f56321 #47773 — [ROCm] Cache fp32 upcast of static e8m0 weight scale in AITER scaled_mm (#47773)
  • 作者: jiacao-amd | +16/-8 | 1 个文件

  The e8m0 weight scale in the AITER block-scaled GEMM path is static, but apply_block_scaled_mm re-runs the «23 bit-shift fp32 upcast plus .contiguous() on it every decode step, for every layer, even though the value never changes. On DeepSeek-V4 FP4 (MI355X / gfx950) this shows up in profiles as ~3.5% of GPU time split across aten::lshift and direct_copy. This PR overrides process_weights…

• ba702e9 #48407 — [Attention] Skip sparse indexer scoring for dense short prefills (#48407)
  • 作者: Yiliu Dong | +230/-12 | 7 个文件

  Try extend #47327 to see some improvement. This PR removes dead indexer scoring work before dense attention. ## Test Result ISL=1024, OSL=1, concurrency=1, 128 requests: | Metric | Base (e6d1310b) | PR (46e170b7) | Delta | |—|—:|—:|—| | Request throughput | 5.998 req/s | 6.150 req/s | +2.54% | | Median TTFT | 167.43 ms | 163.22 ms | −2.52% | | P95 TTFT | 168.76 ms | 164.21 ms |…

• 4fb483c #45432 — [Docs] Expand llm-d integration page (#45432)
  • 作者: IBRAHIM IBRAHIM | +34/-2 | 1 个文件

  Expands the llm-d integration page from a two-sentence blurb into a short overview of what llm-d adds on top of a vLLM deployment. The current page only links out to llm-d without explaining what it does. This update gives readers enough context to know whether llm-d fits their use case before clicking through, while keeping it concise and link-forward. ### What’s added - **What llm-d adds to vLLM…

• 0d0504b #49903 — [Core] Warm up runner-owned Triton kernels before the first request (#49903)
  • 作者: Nick Hill | +183/-189 | 7 个文件

  Several runner-owned Triton kernels only compiled once a real request arrived, spiking first-token latency. Each takes a runtime (non-constexpr) integer that Triton specializes into == 1, % 16 == 0 and neither, so warming a single shape was not enough. - _zero_kv_blocks_kernel is driven by the scheduler’s new_block_ids_to_zero, which no dummy or warmup step reaches. Add KVBlockZeroer.warmup() cove…

• 601fa9a #49612 — [KV Connector] Support NIXL heterogeneous P/D block sizes for hybrid models (#49612)
  • 作者: Nick Hill | +1014/-111 | 7 个文件

  Hybrid (mamba) models previously asserted out heterogeneous block sizes entirely — yet they are the models where P/D block sizes most readily diverge, since the mamba-padded attention block size varies with TP sharding (#41037). Lift the restriction: - Mamba state blocks are indivisible, so their descriptors always use local page geometry and their desc ids are never ratio-expanded; attention desc…

• 9b9fc40 #45841 — add epilogue hook to flex attention (#45841)
  • 作者: liangel-02 | +9/-0 | 1 个文件

  adding optional post-attention epilogue transform to flex attention (this is an example of how someone could use this)

• 98e91a9 #49345 — [PD][NixlPush] Skip extra add_remote_agent step in D->P handshake (#49345)
  • 作者: Nicolò Lucchesi | +6/-3 | 2 个文件

  Address S4 in https://github.com/vllm-project/vllm/issues/48633. In the workflow described here https://docs.vllm.ai/en/stable/design/nixl_kv_push_connector , when D handshakes with P with the purpose of forwarding block_ids for P to write to, D executes a “full handshake”, which includes exchanging kv topology and addresses to register local/remote descs here This is not actually needed to exchan…

• 35efdf6 #47288 — [Elastic EP] Async preparation (#47288)
  • 作者: Itay Alroy | +701/-677 | 21 个文件

  Async Elastic EP preparation This PR moves Elastic EP preparation off the main serving thread. Creating standby process groups, setting up EP communication, staging quant methods, transferring model weights, synchronizing KV-cache memory, and preparing new workers now happen on background worker threads. Requests continue to be served regularly during this work and are blocked briefly only for t…

• d2bfc6f #50103 — [Build] Fix DeepEP CUDA driver stub linking (#50103)
  • 作者: Kevin H. Luu | +15/-0 | 1 个文件

  • discover the CUDA toolkit’s libcuda.so driver stub before building DeepEP - prepend its architecture-specific directory to LIBRARY_PATH - fail early with a clear message if the toolkit stub is missing ## Root cause #45321 updated the pinned DeepEP revision to d4f41e4e93. That DeepEP revision adds -lcuda when linking its extension, but the manylinux wheel builders do not contain a real NVIDIA dri…

• 912d6b6 #47494 — [Rust Frontend] Align sampling validation with Python (#47494)
  • 作者: Reid | +260/-2 | 6 个文件

  Align OpenAI sampling parameter validation with Python. The Rust frontend did not validate several /v1/completions sampling parameters before forwarding requests to the engine. Invalid values such as temperature=5.0, top_p=0.0, min_p=2.0, or frequency_penalty=100.0 could reach Python engine decoding and fail there instead of returning a frontend 400 Bad Request. This PR adds Python-compatible vali…

• bf9f230 #49496 — [Rust Frontend] Fix finish reason for named tool choices (#49496)
  • 作者: Reid | +88/-17 | 3 个文件

  The Rust chat completions path returned finish_reason: “tool_calls” whenever a response contained a tool call. For a forced named function choice, the expected finish reason is “stop”. The previous behavior caused the Rust and Python frontends to return different terminal reasons for the same request in both streaming and non-streaming modes. Preserve whether the request uses a named tool choice w…

• 5ed3faa #49992 — [Rust Frontend] Add ordinary-text tokenizer encoding (#49992)
  • 作者: Bugen Zhao | +448/-2 | 10 个文件

  Segment-aware prompt renderers need separate encoding paths for trusted structural markers and literal text. encode_ordinary keeps literal text on the base tokenizer pipeline even when its spelling matches an added token. This PR adds Tokenizer::encode_ordinary, equivalent to encode(text, false) with every added, special, and control-token matcher bypassed. For tiktoken, this directly delegates to…

• 61ac368 #50090 — [Kimi-K3] Add AttnRes kernels (#50090)
  • 作者: Thien Tran | +1719/-0 | 14 个文件

  #50000. Add AttnRes kernels - Specialized CUDA C++ kernel for NVIDIA sm100 - Triton for NVIDIA fallback and AMD This PR also adds Buildkite CI for K3 ## Test Result —

🖥️ Kernel

• f51193b #49291 — [Kernel][Mamba] Fused-kernel support for align-mode DS-conv state migration with num_accepted_tokens > 1 (#49291)
  • 作者: sungsoo ha | +409/-64 | 5 个文件

  Hybrid/Mamba models support an align-mode Mamba cache path where the running Mamba state is migrated at scheduler-step / block boundaries. For DS conv-state layout, the align path has an intentionally guarded, not-yet-implemented cell: accepted-token tail copies with num_accepted_tokens > 1 (MTP degree > 1). The scalar get_conv_copy_spec helper rejects this case: Root cause. DS conv-state …

🔩 Misc

• df2735e #50092 — [Misc][Minimax-M3]add default video_processor (#50092)
  • 作者: rongfu.leng | +44/-7 | 2 个文件

  Add MiniMaxM3VLVideoProcessor to minimax-m3 Automatically select registered video loader for VLM. ## Test Result —

🦀 Rust Frontend

• 6370e53 #48145 — [Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145)
  • 作者: Seiji Eicher | +198/-13 | 6 个文件

  Why In prefill and decode disaggregation, the prefill stage renders the prompt from messages and tokenizes it. The router forwards the same chat request to the decode stage, which renders and tokenizes it a second time. For long prompts, that repeated render and tokenize adds latency on the decode critical path. The decode stage does not need to redo it. The prefill response already contains th…

• 58f9659 #49665 — [Frontend][Core] Standardize request error handling with VLLMError hierarchy (#49665)
  • 作者: Zach Zhu | +375/-253 | 46 个文件

  This PR implements Steps 1–4 of RFC #48227: - Introduce VLLMError / VLLMClientError / VLLMServerError and reparent existing client/server exceptions under this hierarchy. - Migrate engine/request validation ValueErrors in sampling_params, pooling_params, input_processor, and async_llm to VLLMValidationError. - Catch VLLMClientError (instead of bare ValueError) in AsyncLLM.generate / pooling so cli…

• 01661cc #50081 — [Rust][Benchmark] Make vllm bench serve Rust delegation opt-in (#50081)
  • 作者: Bugen Zhao | +61/-76 | 8 个文件

  Restore the Python implementation as the default for vllm bench serve while the Rust benchmark CLI compatibility gaps are addressed. #48930 automatically delegated supported benchmark configurations to the packaged vllm-rs binary. However, the Python and Rust implementations currently still differ in accepted arguments, underscore aliases, defaults, and help output. For example, Python accepts –r…