共 48 个 commit,涉及 284 个文件,+9206/-5773 行变动。
概要
| 统计项 | 数值 |
|---|---|
| Commit 数 | 48 |
| 变更文件 | 284 |
| 新增行数 | +9206 |
| 删除行数 | -5773 |
Commit 列表
🐛 Bug Fix
- cad4ca1 #44663 — [Bugfix] Add X-Session-ID from conversation_id in multi-turn benchmark (#44663)
- 作者: Tyko Niemi | +2/-0 | 1 个文件
Set X-Session-ID in send_request when conversation_id is present. Keep the header unset when conversation_id is not provided. Co-authored-by: GitHub Copilot Set the X-Session-ID HTTP header from conversation_id in the multi-turn benchmark client (send_request) when a conversation_id is present. The header is left unset when no conversation_id is provided, preserving existing behavior. This enables…
- dc10e46 #44983 — [Bugfix] Fix minimax_qk_norm_fusion (#44983)
- 作者: Jee Jee Li | +41/-9 | 1 个文件
- Fixes https://github.com/vllm-project/vllm/issues/44003 - Fix the fallback of minimax_qk_norm_fusion when VLLM_ALLREDUCE_USE_FLASHINFER=1 is set, which raises the error below. This is caused by the variance in minimax_qk_norm being fp32, which isn’t compatible with FlashInfer’s AllReduce. ## Test Result —
- 6690a0c #44629 — [PD][Bugfix] Fix KV Cache sharing with HMA (#44629)
- 作者: Nicolò Lucchesi | +38/-12 | 3 个文件
HMA is now the default way to serve models even when a connector is provided as of https://github.com/vllm-project/vllm/pull/41847. Unfortunately we missed covering an important feature that is layers that share the same KV cache tensor, which currently crashes on main when attempting to fetch the corresponding layer_spec. The fix is straightforward, as we ensure that layers that do not need a kv …
- dab60fc #44903 — [Bugfix][CI] Fix
test_offloading_connector.py::test_fs_tiering_offloading(#44903)- 作者: Nicolò Lucchesi | +1/-1 | 1 个文件
Fix as per https://buildkite.com/vllm/ci/builds/70519/canvas?jid=019ea73c-9bc3-44be-b07e-407911b09a14&tab=output by increaasing max model len
- e6fc848 #43844 — [Bugfix][MiniCPM-o] Fix cuda/cpu device mismatch in Resampler2_5 pos_embed (#43844)
- 作者: Parth Ashwin Jain | +3/-1 | 1 个文件
Resampler2_5.forward adds the cached positional embedding to the input without moving it to the input’s device: When self.pos_embed (a non-persistent buffer) stays on CPU while the hidden states x are on the GPU, this raises: ## Root cause pos_embed is registered as a non-persistent buffer, so it is not moved by the normal weight-loading device placement and stays on CPU unless something explicitl…
- f843ac1 #44952 — [Bugfix][CI] Gemma3 Transformers multimodal encoder profiling and build prompt-embedding fixtures (#44952)
- 作者: Andreas Karatzas | +35/-2 | 2 个文件
This PR: - Bugfixes Gemma3 Transformers multimodal encoder profiling to use Gemma3’s vision patch-token scale. - Builds prompt-embedding fixtures (gated on ROCm only) before starting the API server so HF cleanup is not blocked by vLLM-owned GPU memory. The Gemma3 failure showed up as a ROCm GPU hang in the multimodal shard. The failing log used max_num_seqs=256 and profiled 64 image items of the m…
- ebf53ba #44729 — [Bugfix][Rust Frontend] Set a structured-output backend so requests do not 500 (#44729)
- 作者: Ting SUN | +40/-0 | 1 个文件
With VLLM_USE_RUST_FRONTEND=1, every structured-output request (response_format json_schema/json_object, structured_outputs regex/choice) returns HTTP 500: the engine raises Unsupported structured output backend: None in grammar_init. The backend is normally resolved by SamplingParams._validate_structured_outputs in the Python Processor; the Rust frontend sends the EngineCoreRequest straight to En…
- baacbfc #42978 — [ROCm][MLA][Bugfix] Reserve FP8 prefill workspace before lock for Kimi-K2.5 (#42978)
- 作者: xaguilar-amd | +34/-5 | 1 个文件
Fixes a ROCm AITER dense MLA FP8 prefill crash after #42509 where the per-call scratch workspace introduced for mla_prefill_ps_asm_fwd could be first requested after the global workspace manager had already been locked. The fix reserves the maximum FP8 prefill scratch shapes while initializing the FP8 persistent-scheduling metadata buffers, before lock_workspace() runs. The reservation is sized fr…
- d8218b1 #44750 — [Bugfix] Propagate ImportError from load_audio_pyav when vllm[audio] … (#44750)
- 作者: littlecircle0730 | +1/-1 | 1 个文件
Fix #44746. When vllm[audio] extras are not installed, av is a PlaceholderModule whose attribute access raises ImportError(“Please install vllm[audio] for audio support”). In load_audio_pyav(), this ImportError was caught by the broad except Exception clause and silently converted to a generic ValueError, making the helpful install hint unreachable. PR #39473 added except ImportError: raise in the…
- 540aaf2 #44264 — [Bugfix][Model] Qwen3-Omni: move cu_seqlens to GPU before VIT attention (#44264)
- 作者: Lanze Liu | +3/-0 | 1 个文件
Fixes #44180. During profile_run the multimodal grid_thw arrives on CPU, so the cu_seqlens tensor built from it inherits the CPU device. Passing this CPU tensor into the FA3 vit attention path raises RuntimeError: cu_seqlens_q must be on CUDA and crashes engine init when loading Qwen/Qwen3-Omni-30B-A3B-Thinking (and the Instruct variant). Move cu_seqlens to self.device after construction so the FA…
- 6afa250 #44735 — [Bugfix] Canonicalize FP8 weight layout to (K, N) at the source (#44735)
- 作者: Michael Goin | +11/-29 | 3 个文件
Fixes #44110 at the root cause, as an alternative to #44113. MarlinFP8ScaledMMLinearKernel.process_weights_after_loading previously tried to detect whether its incoming weight was (N, K) or (K, N) and transpose accordingly. The shape-based heuristic was a no-op when N == K, silently corrupting square layers. #44113 swaps the heuristic from shape to is_contiguous(), but that just trades one fragile…
- 823a0ab #44897 — [Bugfix][MoE] Fix fused MoE expert mapping helper call sites (#44897)
- 作者: Mohammad Miadh Angkad | +26/-11 | 6 个文件
Fixes stale fused MoE expert mapping call sites after #41184, restoring pre-commit on main and avoiding runtime failures in affected weight-loading paths. CI
🧪 CI/Tests
- b697119 #44040 — [ROCm][CI] Stabilize ModernBERT token-classification parity against Hugging Face (#44040)
- 作者: Andreas Karatzas | +21/-6 | 1 个文件
Stabilizes flakiness of: with the representative mismatch: The test runs disham993/electrical-ner-ModernBERT-base with float precision and compares vLLM token-classification probabilities against Hugging Face. vLLM’s ModernBERT implementation had a few places where it did not follow the model config / HF implementation: 1. RoPE was always built with dtype=torch.float16, even when the model was req…
- b12e42d #44481 — [XPU][CI] Refine docker image build and pull/create lock mechanism in Intel GPU CI (#44481)
- 作者: xiangdong | +86/-87 | 2 个文件
- Reduce XPU Docker image size and layer bloat - Unify Intel Docker pull+create under one lock ## Test Result —
- 2385e14 #43022 — [ROCm][CI] Stabilize sleep-mode memory release (#43022)
- 作者: Andreas Karatzas | +25/-1 | 1 个文件
- Free and re-reserve the ROCm virtual address only in the sleep-mode unmap path. - Keep wake-up compatible by re-reserving the same virtual address. - Synchronize sleep/wake transitions around release and remap. test_sleep_mode intermittently failed on MI300 with a HIP OOM while the GPU reported almost all VRAM free. ROCm was releasing physical handles, but the reserved virtual address range kept…
- 996222f #44947 — [CI] Reorganize entrypoints CI (#44947)
- 作者: wang.yuqi | +125/-124 | 20 个文件
- Entrypoints Unit Tests 7m 43s - tests/entrypoints/unit_tests - tests/entrypoints/weight_transfer - Entrypoints Integration (LLM) 40m 49s - tests/entrypoints/llm - Entrypoints Integration (API Server) 33m 44s - tests/entrypoints/serve - Entrypoints Integration (API Server OpenAI - Part 1) 45m 58s - pytest -v -s entrypoints/openai –ignore=entrypoints/openai/completion –ignore=entrypoints/openai/…
- d3de615 #44940 — [XPU][CI] fix test case path (#44940)
- 作者: Kunshang Ji | +1/-1 | 1 个文件
fix issue after #44819 Intel CI ## Test Result —
- 05cb606 #44809 — [ROCm][CI] Re-route NixlConnector jobs (#44809)
- 作者: Andreas Karatzas | +15/-11 | 2 个文件
Re-route ROCm NixlConnector CI jobs that use KV connectors through TRITON_ATTN instead of forcing ROCM_ATTN. This updates the AMD Buildkite pipeline entries for the NixlConnector accuracy and spec decode acceptance jobs on MI250, MI300, and MI355, and teaches the config sweep helper to accept a generic ATTENTION_BACKEND=… override. ## Changed Test Groups - MI250: NixlConnector PD + Spec Decode a…
- 93ee4cd #44819 — [CI] Consolidate multimodal entrypoint tests. (#44819)
- 作者: wang.yuqi | +107/-62 | 24 个文件
Consolidate multimodal entrypoint tests. - These tests have more in common. - Reduce the number of files in the tests/entrypoints/openai/chat_completion folder. These all make the entrypoint tests easy to maintain. tests/entrypoints/multimodal/ ## Test Result —
📦 Other
- b4c6dc6 #42262 — [WIP][XPU] upgrade torch-xpu to 2.12 (#42262)
- 作者: Kunshang Ji | +9/-9 | 5 个文件
Test Result —
- 2ee5106 #39425 — Remove
raw_inputsfrom transformers backend (#39425)- 作者: Raushan Turganbay | +3/-12 | 1 个文件
Transformers backend models doesn’t need anymore to pass arbitrary kwargs to model’s forward, specifically token-types. vLLM’s attention backend infers multimodal toke types at runtime and computes correct mask based on config.model_type Verified that the backend also goes through the “bidirectional image mask” path with correct mm_range values
- ee4d7df #44907 — [Cohere] Cohere2 moe parser fix (#44907)
- 作者: Terrence Zhao | +33/-12 | 1 个文件
- Normalize Cohere Command streaming tool-call deltas so Melody placeholder fields are omitted from OpenAI-compatible responses. - Preserve pending tool-call IDs until the first real function-name delta, preventing clients from seeing invalid empty tool names. - Update Cohere2 MoE weight loading to use the current fused_moe_make_expert_params_mapping helper after the FusedMoE API change. ## Test R…
- 3e8afdf #44747 — [Cohere] Fix Cohere2MoE weight loading when using Transformers ≥5.10 (#44747)
- 作者: Terrence Zhao | +27/-9 | 1 个文件
Problem Cohere2MoeDecoderLayer and Cohere2MoeAttention relied on getattr(config, “first_k_dense_replace”, 0) to decide whether a layer uses a dense MLP or MoE. Under Transformers 5.10+, Cohere2MoeConfig consumes first_k_dense_replace and sets mlp_layer_types = [“dense”, “sparse”, …], leaving first_k_dense_replace as None. vLLM then defaulted to 0, built layer 0 as MoE, and failed to load chec…
- 1c23c42 #44901 — [Rust Frontend] Support Kimi K2 tool call IDs (#44901)
- 作者: Maria Guevara | +147/-8 | 5 个文件
Preserve Kimi K2 model-generated tool-call IDs in the Rust frontend output path , adds an optional ToolParser::tool_call_id() hook. Existing parsers keep the default None behavior and still use generated call_
values. - Kimi K2 parser stores the raw header ID and exposes it through that hook, so streamed and final OpenAI-compatible ool calls preserve IDs like functions.{name}:{index}. - This m… - 69fdaff #44222 — [Rust Frontend] Add /tokenize and /detokenize endpoints (#44222)
- 作者: TanNgocDo | +836/-39 | 9 个文件
Adds POST /tokenize and POST /detokenize (being tracked in https://github.com/vllm-project/vllm/issues/44280) to the Rust server, matching the Python OpenAI server’s root-path endpoints. Encoding/decoding runs entirely in-process via DynTokenizer — the inference engine is not involved. - /tokenize accepts both forms (serde untagged): - Completion: encodes the raw prompt string (no chat tem…
- 80e2c44 #42864 — [ROCm][Compile] Fuse AR + RMSNorm + per-group FP8 quant (+ DSv3.2 indexer fan-out) (#42864)
- 作者: Markus Hartikainen | +795/-2 | 3 个文件
Two new patterns in RocmAiterAllReduceFusionPass that fuse the all_reduce -> RMSNorm[+add] -> per-group FP8 quant [-> bf16 indexer GEMM] chain into one AITER call, eliminating ~535 us / decode step of standalone triton_per_token_group_quant_fp8 launches on DeepSeek V3.2 MI355X TP4. This is the AR-side analogue of #41825 (which fixed the quant-only half) and the ROCm port of the flashinfer AllReduc…
- 5b3807e #42892 — [KV Events] Switch event structs from array to map encoding (#42892)
- 作者: Sage | +1/-4 | 2 个文件
Context KV cache events are published over ZMQ as msgpack-encoded batches consumed by external subscribers (routing daemons, cache managers, etc.). The encoding used a positional array format — every field identified purely by its index — which makes every schema addition a potential break for subscribers that haven’t been updated at the same time. This was first flagged by @njhill in #27577, …
- d841386 #44321 — [Rust Frontend] Support API key authentication (#44321)
- 作者: Chao-Ju Chen | +384/-15 | 14 个文件
This PR adds Rust frontend support for OpenAI-compatible API key authentication through: - –api-key, accepted multiple times - api_key in the Python-supervised Rust frontend –args-json contract, accepting either a string or a list of strings - VLLM_API_KEY fallback when no explicit –api-key / api_key value is provided The Rust server now installs an authentication middleware when API keys are c…
- fff9210 #44918 — [CI/Docs] Remove stale disagg prefill links (#44918)
- 作者: Mohammad Miadh Angkad | +0/-4 | 1 个文件
PR #44854 removed the old disaggregated prefill example and benchmark files, but docs/features/disagg_prefill.md still linked to them. This removes those stale docs references and fixes the Read the Docs failure on main. cc @NickLucche @hmellor @youkaichao
- 70db148 #44144 — [DSV4][XPU] Add MHC fused_post_pre support (#44144)
- 作者: Ma Jian | +112/-17 | 2 个文件
Add MHCFusedPostPreOp XPU support for DeepSeek-V4 on Intel XPU, enabling the fused MHC post+pre path in the decoder loop (matching the AMD/CUDA pattern). - vllm/model_executor/layers/mhc.py: Implement forward_native for MHCFusedPostPreOp (decomposes into mhc_post_torch + mhc_pre_torch); add forward_xpu delegating to forward_native. - vllm/models/deepseek_v4/xpu/model.py: Update decoder loo…
- 7c2aa31 #43595 — fix: prevent MM cache hang from stale LRU order keys (#43595)
- 作者: jeff.ye | +1/-3 | 1 个文件
FIX #43941 LRUCache.touch() inserted keys into the internal LRU order even when the key was not present in the cache data. The multimodal processor and receiver caches touch every hash in a request before updating the cache so that items used by the current request are not evicted midway through the batch. For cache misses, the old touch() behavior created order-only ghost keys. When the cache was…
- 4128605 #44929 — [Docs] Remove broken link to deleted disaggregated_prefill.sh (#44929)
- 作者: Lanze Liu | +0/-2 | 1 个文件
disaggregated_prefill.sh was removed by #44854 but the reference in docs/features/disagg_prefill.md was not updated. This causes MkDocs to abort in –strict mode, breaking the RTD build for all PRs. The per-connector examples already below that line serve as usage guidance. Verified docs/features/disagg_prefill.md no longer references the deleted file. RTD build passes once the broken link is remo…
- e2f993d #41183 — [WideEP] Integrate DeepEP v2 (#41183)
- 作者: Tyler Michael Smith | +1409/-17 | 22 个文件
Integrating https://github.com/deepseek-ai/DeepEP/pull/605 Unit tests are passing, and it is working e2e. GSM8k appears to be good and will follow up with more thorough e2e tests. Notes: * I couldn’t get this working on an 8xB200 system as DeepEP v2’s ElasticBuffer unconditionally asserts NCCL GIN availability even for intra-node NVLink-only. This is a TODO. * This requires NCCL 2.30.4, and PyTorc…
- 2c27c29 #44450 — [Model Runner V2] Fix mrv2 mm lora issue (#44450)
- 作者: Wentao Ye | +183/-0 | 3 个文件
VLLM_USE_V2_MODEL_RUNNER=1 pytest tests/lora/test_qwenvl.py::test_qwen3vl_vision_lora -xvs Now
- ba94a3b #40470 — [Attention] Extract KV-cache update from CPU attention backend (#40470)
- 作者: Diego Maniloff | +31/-22 | 1 个文件
Part of #32335 - Extract the KV-cache update from CPUAttentionBackendImpl.forward() into a separate do_kv_cache_update() method, as part of issue #32335. - Set forward_includes_kv_cache_update = False on CPUAttentionBackend so the attention layer calls do_kv_cache_update() before forward(). - The CPU backend uses ops.cpu_attn_reshape_and_cache (not reshape_and_cache_flash), which requires an ISA p…
- dc68bd8 #41184 — [MoE Refactor] FusedMoE/MoERunner inversion refactor (#41184)
- 作者: bnellnm | +2734/-2027 | 90 个文件
Invert the MoERunner <-> FusedMoE relationship The MoERunner will own the FusedMoE (renamed to RoutedExperts) The FusedMoE class will go away. Some model weight loading code needed updating since the paths for MoE weights now has an extra level, e.g. .experts.
is now .experts.routed_experts. Based on the following PRs: cc @yzong-rh CI + MoE refactoring tests Run all MoE layer tests (incl… - 753e9d5 #44132 — [Quantization] add online fp8 ptpc (#44132)
- 作者: Walter Beller-Morales | +385/-8 | 6 个文件
Add online fp8 per-token activation + per-channel weight (ptpc) quantization Added a unit test tests/quantization/test_fp8_ptpc.py and a quality test tests/models/quantization/test_fp8_ptpc.py ## Test Result ✅ —
- ac3409d #44708 — [Benchmark] Auto-detect and correct client/server tokenizer mismatch for random dataset (#44708)
- 作者: akii96 | +81/-1 | 1 个文件
Alternative to #42532. Addresses the same problem (input token inflation when bench-side and server-side tokenizers disagree) but takes a different approach based on reviewer feedback there: - Zero changes to dataset code (@DarkLight1337 ’s main concern) - No new CLI flags - catches any model/tokenizer version mismatch, not just the current DeepSeek-V3.2 case After get_samples(), probes the server…
- 980796c #44852 — [CI/Build][CPU] Fix flaky CI image build failure and unexpected warnings (#44852)
- 作者: Li, Jiang | +5/-4 | 2 个文件
- Add cache lock to Rust build stage to avoid contention between building on a same host - Add reference to to avoid GC collects it too early ## Test Result —
- 5add018 #44854 — [Connector] Remove
P2pNcclConnector(#44854)- 作者: Nicolò Lucchesi | +2/-3069 | 18 个文件
After some more recent dicussions, for the same reasons listed here https://github.com/vllm-project/vllm/issues/33115, we’ve decided to remove the P2pNcclConnector. I would still like to thank @Abatom and everyone who was contributed to the connector as this helped PD take off the ground in its early stages. Mind that similar connectors can still be implemented OOT leveraging the current Connector…
- d5fe994 #44419 — [CPU][Spec Decode] Warn about throughput loss when libiomp5 is not preloaded (#44419)
- 作者: Jonathan Mamou | +13/-2 | 1 个文件
Add an explicit warning when speculative decoding is enabled on CPU without libiomp5 in LD_PRELOAD. Without Intel’s OpenMP runtime, SD throughput drops by ~2× due to the overhead of GNU libgomp in the frequent short parallel regions that speculative decoding uses. While investigating issue #44191, we discovered that the performance gap between the official Docker image and source builds is entirel…
- fa662b1 #44470 — [XPU] Cap topk/topp Triton BLOCK_SIZE to 4096 to fix Top-p mask difference failures (#44470)
- 作者: Chaojun Zhang | +6/-1 | 2 个文件
Cap the Triton BLOCK_SIZE to 4096 in topk_topp_triton.py on XPU to fix Top-p mask difference too large test failures. - vllm/v1/sample/ops/topk_topp_triton.py: limit XPU block size to 4096 for deterministic sampling - .buildkite/intel_jobs/misc_intel.yaml: enable test_topk_topp_sampler.py on XPU CI
- 3c0b443 #44499 — [Rust Frontend] Add /pause, /resume, /is_paused endpoints (#44499)
- 作者: Sahil Singh | +284/-0 | 4 个文件
Adds POST /pause, POST /resume, and GET /is_paused to the Rust frontend, contributing to “RL / admin / lifecycle APIs” in the Rust frontend roadmap (#44280). | Route | Utility call | Args | Response | |—|—|—|—| | POST /pause | pause_scheduler | (mode, clear_cache) | {“status”: “paused”} | | POST /resume | resume_scheduler | () | {“status”: “resumed”} | | GET /is_paused | is_scheduler_pause…
⚡ Performance
- 7a89b72 #44176 — [Perf] fuse qk rmsnorm rope gate for qwen3.5 (#44176)
- 作者: Jiangyun Zhu | +418/-12 | 4 个文件
combine split + QK-RMSNorm + partial RoPE + gate copy into one kernel launch Ref: https://github.com/lightseekorg/tokenspeed/pull/228 ## Test Result main PR ## Accuracy main |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |—–|——:|—————-|—–:|———–|—|—–:|—|—–:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.6702|± |0.0129| | | |strict-match | 5|exact_ma…
- 59401ac #44830 — [Kernel][Perf] Tune fused_moe FP8 config for Qwen3-Next-80B tp=4 on H100 (+25% at batch 96-512) (#44830)
- 作者: Qiuyang Yue | +43/-0 | 1 个文件
Add a tuned Triton fused_moe config for Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 on a single NVIDIA H100 80GB HBM3 with tp=4 and FP8 w8a8 blockwise quantization (block_shape=[128, 128]) Qwen3-Next-80B has num_experts=512, num_experts_per_tok=10, moe_intermediate_size=512. With tp=4 this gives the kernel shape E=512, N=128. The current get_default_config in fused_moe.py is ~25% slower than the tuned co…
- 9f153aa #40576 — [MM][Perf][CG] Support ViT full CUDA graph for glm4_1v image and video inference (#40576)
- 作者: GuorongYe | +480/-25 | 4 个文件
Following https://github.com/vllm-project/vllm/issues/38175, this PR implements ViT CUDA graph support for glm4_1v models image and video inference . The implementation draws references from https://github.com/vllm-project/vllm/pull/35963 (image) and https://github.com/vllm-project/vllm/pull/38061 (video) 1. Functional Test 2. Benchmark in some scenarios: - no DP VIT + eager vs no DP VIT + cuda gr…
🔩 Misc
- 3f627eb #44595 — [Misc] usage_stats: report more engine, spec-decode, and EP config (#44595)
- 作者: Zachary Xi | +44/-4 | 1 个文件
Added fields to usage_stats so we can better understand how people use vLLM in production and which fields to focus efforts on. Fields added: - Batching limits: max_model_len, max_num_seqs, max_num_batched_tokens - Spec decoding spec_decode_method, num_speculative_tokens - Wide expert parallel: enable_eplb, num_redundant_experts, num_experts - Backend: attention_backend - Torch details: compilatio…
🔧 Refactor
- bc941f3 #44856 — [Rust Frontend] [Refactor] Refine utility call interfaces (#44856)
- 作者: Bugen Zhao | +125/-82 | 5 个文件
Minor refinements on Rust frontend utility-call interfaces. Follow-up of #44499. - Add a protocol-owned PauseMode enum for abort / wait / keep, and use it for sleep and pause_scheduler instead of passing raw strings. - Reuse a call_utility_consensus helper for utility calls whose results must agree across engines, such as is_sleeping and is_scheduler_paused, while keeping operation-style boolean u…