共 89 个 commit，涉及 519 个文件，+29842/-10714 行变动。
概要
| 统计项 | 数值 |
|---|---|
| Commit 数 | 89 |
| 变更文件 | 519 |
| 新增行数 | +29842 |
| 删除行数 | -10714 |
Commit 列表
📦 Other
- e840f0d #47140 — [Platform] Replace
torch.cuda.Eventwithtorch.Event(#47140)- 作者: Kunshang Ji | +60/-70 | 31 个文件
part of https://github.com/vllm-project/vllm/issues/30679 CI ## Test Result —
- d8f483d #46301 — [Spec Decode] Fix hidden-state extraction block size for hybrid verifiers (#46301)
- 作者: Igor Margulis | +187/-15 | 3 个文件
Purpose extract_hidden_states + ExampleHiddenStatesConnector silently produces all-zero hidden states on hybrid (Mamba + attention) verifiers such as Qwen3.5-35B-A3B, and crashes with a CUDA index out of bounds at scale. Training on the zero/garbage states “converges” (validation looks fine) but the resulting draft model has ~0% real acceptance — see vllm-project/speculators#613. This supe…
- c231d1f #47007 — fix(security): bound tokenizer work when explicit truncation_side is set (#47007)
- 作者: Juan Pérez de Algaba | +94/-9 | 2 个文件
When both truncation_side and truncate_prompt_tokens are set, get_encode_kwargs() disabled tokenizer-level truncation entirely, allowing attacker-controlled prompts to force unbounded tokenization before post-tokenization slicing. Restore a tokenizer-level max_length bound (context window + 1) in the explicit-side branch, and add a defense-in-depth character-level pre-truncation in _text_len_check…
- db808b3 #46781 — [Model Runner V2][Spec Decode] Implement block verification for rejection sampling (#46781)
- 作者: Giancarlo Delfin | +641/-85 | 4 个文件
Context Block verification as described in https://arxiv.org/pdf/2403.10444v3 is an approach to rejection sampling that is expected to produce acceptance rates at least as high as standard (per-token) rejection sampling, for a comparable cost. This feature is currently not supported in Model Runner V2, but is in progress for V1. The main idea of block verification is that we look at the joint pr…
- 7a327f0 #47125 — [Rust Frontend] Simplify unit tests with shared
TestTokenizer(#47125)- 作者: Bugen Zhao | +682/-828 | 28 个文件
Signed-off-by: Bugen Zhao i@bugenzhao.com Introduce a shared, general & configurable TestTokenizer for Rust frontend tests, which encodes and decodes UTF-8 at the byte level, with pluggable multi-character special tokens. Migrate most ad-hoc tokenizer stubs to it, reducing maintenance burden as the codebase grows. This also tightens the Tokenizer trait by requiring id_to_token implementations, s…
- 1ab9522 #47058 — Remove more unnecessary
load_weightsmethods (#47058)- 作者: Harry Mellor | +1028/-3778 | 63 个文件
- FusedMoE now takes ckpt_names instead of expert_mapping and stores them individually in RoutedExperts - RoutedExperts.load_weights then calls RoutedExperts.get_expert_mapping which uses these names to construct the expert mappings on the fly - RoutedExperts.load_weights now handles checkponts like Qwen3 VL MoE where the fused expert weights may be transposed - RoutedExperts.load_weights now brea…
- 0fc2512 #46450 — [KV Offload] Pass
ScheduleEndContexttoon_schedule_endhook (#46450)- 作者: Ronen Schaffer | +42/-14 | 11 个文件
- Extend on_schedule_end() to accept a ScheduleEndContext dataclass containing new_req_ids and preempted_req_ids, enabling managers to react to per-step scheduling decisions - The new dataclass uses Iterable[str] field types for flexibility - Updates both OffloadingManager and SecondaryTierManager base classes and all implementations. ## Test Result —
- 62c7d80 #47151 — Forward fix nightly errors from #44589 (#47151)
- 作者: Harry Mellor | +27/-17 | 3 个文件
This PR forward fixes the failures introduced to main by #44589. Supersedes #47096. Fixes for Language Models Test (MTEB): - Move the hf_to_vllm_mapper so it belongs to Gemma3Model so that it can load itself. This is necessary because Gemma3Model is registered directly as Gemma3TextModel - Reinstate the prefixing of model. to JinaEmbeddingsV5Model weights so that it can load correctly Fix for Mult…
- ab80b3d #47139 — [CI/Build] Bump PyNvVideoCodec version (#47139)
- 作者: Isotr0py | +1/-1 | 1 个文件
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED. - Fix https://github.com/vllm-project/vllm/pull/44465#issuecomment-4841638946 ## Test Result — BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)
- 91055ef #47134 — [XPU] C++ implementation for get_memory_info (#47134)
- 作者: Qiming Zhang | +72/-0 | 1 个文件
Torch.xpu.mem_get_info returns wrong memory info with latest UMD. More details please refer to https://github.com/vllm-project/vllm-xpu-kernels/blob/70387bfdcc15f85628ed31b8faf62af537096435/tests/test_get_memory_info.py#L16 Use API of kernels to replace it. Refer to https://github.com/jikunshang/vllm/commit/bd4f96f97fa03cf70dda72ac9098b72f3fdb8477. Add some bounds and type check.
- 3675bcf #47101 — [Rust Frontend] Refactor TLS serve path with unified
MaybeTlsListener(#47101)- 作者: Bugen Zhao | +138/-180 | 6 个文件
Signed-off-by: Bugen Zhao i@bugenzhao.com Introduce a MaybeTlsListener as the only listener type the server will operate on, which can either be a plain TCP/UDS listener, or a SSL wrapper over that. Implement axum::serve::Listener and Stream<Item = std::io::Result
> for that so it can be used both for axum HTTP server and tonic gRPC server. cc @tahsintunan Cargo tests ## Test Resu… - bdbd727 #47110 — [Rust Frontend] Extend renderer/parser roundtrip tests to support token ids (#47110)
- 作者: Bugen Zhao | +160/-79 | 2 个文件
Some model paths are token-id native: renderers may directly yield token ids, and output processors may directly consume token ids. This PR extends the roundtrip tests to support raw token ids in those cases, demonstrated by the GPT-OSS/Harmony path. ## Test Result All passed locally. I also verified that roundtrip_gpt_oss fails if generated chunk token ids are cleared, which confirms the test cov…
- 5dc36a4 #47143 — [Model] Remove Tarsier, Tarsier2 (#47143)
- 作者: Harry Mellor | +2/-872 | 12 个文件
These models are: - >1.5 years old - See almost no usage in vLLM - Are only downloaded by vLLM CI so the download stats you see on HF are misleading This PR removes them both and adds entries to _PREVIOUSLY_SUPPORTED_MODELS for them. Note that Tarsier2 has a previously supported version of v0.23.0 as that was the last version of vLLM to support Transformers v4 Closes https://github.com/vllm-proje…
- 5360477 #33057 — Bump actions/checkout from 6.0.1 to 7.0.0 (#33057)
- 作者: dependabot[bot] | +2/-2 | 2 个文件
Bumps actions/checkout from 6.0.1 to 7.0.0.
- 8cf7c4d #46020 — [Attention Backend] add HPC-Ops Attention backend (#46020)
- 作者: Cheng Jiang | +978/-14 | 10 个文件
Support hpc-ops attention backend. HPC-Ops is a production-grade, high-performance, and easy-to-use operator library for LLM inference, developed by the Tencent Hunyuan AI Infra team. You can enable hpc-ops attention backend by specify –attention_backend HPC_ATTN –kv-cache-dtype fp8_e4m3 –block-size 64. WARNING: The HPC attention backend currently supports only the Hy3-FP8 model. If you wan…
- 364ee36 #47010 — fix(security): prevent image decompression bomb OOM denial of service (#47010)
- 作者: Juan Pérez de Algaba | +115/-12 | 7 个文件
Add VLLM_MAX_IMAGE_PIXELS env var (default 178,956,970 – matching PIL’s 2x decompression-bomb threshold) and enforce a pixel-count check in ImageMediaIO.load_bytes() before calling image.load(), so raster memory is never allocated for oversized images. Additionally, replace the global Image.MAX_IMAGE_PIXELS = None in the NemotronVL processors with a finite 300M-pixel cap to restore PIL’s built-in…
- 14f8660 #47032 — [CI/Build] Add CPU test dependency pre-commit hooks (#47032)
- 作者: Li, Jiang | +1306/-21 | 3 个文件
- Add CPU test dependency pre-commit hooks to pin package versions with CUDA ## Test Result —
- 8cc2423 #46433 — [XPU] Optimize XPU worker shutdown logic to prevent resource leak (#46433)
- 作者: Chaojun Zhang | +41/-5 | 7 个文件
Purpose: Optimize the XPU shutdown flow to properly release oneCCL/Level Zero resources when vLLM server stops. This also fixes a hang issue where repeated server startups on the same XPU devices would deadlock during oneCCL initialization. ## Problem: With the default shutdown_timeout=0 (abort mode), worker processes are force-killed without executing cleanup code. On XPU, this leaves Level Ze…
- fb42e52 #44825 — [Platform] Replace
torch.cuda.mem_get_infowithtorch.accelerator.get_memory_info(#44825)- 作者: Kunshang Ji | +56/-52 | 17 个文件
part of https://github.com/vllm-project/vllm/issues/30679 CI ## Test Result —
- 0feca7f #46807 — PD disagg with Mooncake Connector: GDN support (Qwen3.5) and MLA support (Deepseek-V4-Flash) (#46807)
- 作者: Dakai An | +735/-103 | 7 个文件
Mooncake P/D disaggregation did not work correctly for hybrid KV-cache models such as Qwen3.5, where one model has both full-attention KV cache groups and GDN state groups represented as MambaSpec. Nixl now has supported GDN pd disagg here https://github.com/vllm-project/vllm/pull/41869, but mooncake connector lacks this implementation. The previous Mooncake path treated request block IDs and regi…
- e45c8a9 #46833 — [Rust Frontend] Start current wave for a stale DP FirstRequest (#46833)
- 作者: Blas Rodriguez Irizar | +84/-9 | 2 个文件
In the in-process DP coordinator, notify_first_request enqueues FirstRequest{wave: W} while engines are paused, but the runner broadcasts START_DP_WAVE later. If engines report WaveComplete(W) in that gap, current_wave advances to W+1. The stale FirstRequest{wave: W} was then applied unconditionally, rewinding current_wave back to W and broadcasting a superseded wave. This matches the Python coord…
- a16dbd5 #47040 — [Rust Frontend] Avoid LoRA registry scans without active LoRA requests (#47040)
- 作者: Reid | +80/-4 | 1 个文件
The Rust frontend updates vllm:lora_requests_info from frontend-side request tracking after each engine output batch. In deployments without LoRA requests, RequestRegistry::lora_adapter_states() still scanned every in-flight request to produce two empty adapter sets. This PR tracks the number of active LoRA requests in RequestRegistry and returns empty adapter sets immediately when the count is ze…
- bec232a #42285 — Secondary tier implementation for PD disaggregation (#42285)
- 作者: liranschour | +7471/-0 | 22 个文件
In this design PD disaggregation is based on the vLLM CPU KV cache which is per vLLM instance and it is in canonical layout (single TP unified block size). The PD Connector is a secondary tier that implements the SecondaryTierManager interface. Orchestration between the primary tier (CPU Manager) and secondary tiers is done by the TieringManager, which is transparent to the secondary tiers. ## Tes…
- b5c9e1a #46740 — [LoRA] Add language-backbone LoRA support for MiniCPM-V 4.6 (#46740)
- 作者: linitra24 | +3/-1 | 2 个文件
This PR adds LoRA support for the language backbone of MiniCPMV4_6ForConditionalGeneration. With this change, MiniCPM-V 4.6 can load PEFT LoRA adapters targeting the language model part through vLLM’s existing LoRA mechanism. This addresses the current error where MiniCPMV4_6ForConditionalGeneration is reported as not supporting LoRA yet. Vision-side LoRA support is not included in this PR and wil…
- ae2c4f3 #46804 — [XPU][UT]Fix xpu pass_config.fuse_norm_quant assert issue (#46804)
- 作者: Yejing Lai | +7/-3 | 2 个文件
Fix some ut meet AssertionError: pass_config.fuse_norm_quant: expected True, got False. Related failed UT: tests/test_config.py::test_vllm_config_defaults[RedHatAI/Qwen3-8B-speculator.eagle3-compilation_config5-2] tests/test_config.py::test_vllm_config_explicit_overrides tests/test_config.py::test_fusion_pass_op_priority
- af1ee8c #44070 — fix(config): reject negative max_logprobs (except -1) and long_prefill_token_threshold (#44070)
- 作者: hcl | +2/-2 | 2 个文件
Fixes #43985. ## what max_logprobs and long_prefill_token_threshold accepted any int, including bare negatives that downstream code didn’t expect. - max_logprobs=-5 → validate_logprobs only rewrites -1 to vocab_size, so other negatives land as the cap and surface as a confusing max allowed: -5 error (or silent no-op when no user logprobs). - long_prefill_token_threshold=-5 → clamp 0 < threshold < …
- b8cb75b #45890 — [Rust Frontend] Add static HTTPS and mTLS support for HTTP and gRPC (#45890)
- 作者: Tahsin Tunan | +1942/-125 | 16 个文件
Summary - Adds TLS termination to the Rust frontend (tracking: #44280): HTTPS and mutual TLS, on both the HTTP and gRPC servers. - Supported: –ssl-certfile, –ssl-keyfile, –ssl-ca-certs, –ssl-cert-reqs (0/1/2), and –ssl-ciphers. A combined cert+key PEM works (the key falls back to the certfile), mirroring uvicorn. - Built on OpenSSL (openssl/tokio-openssl), reusing the OpenSSL already in t…
- 4391689 #46346 — [GDN] Improve kkt kernel of CuteDSL prefill backend (#46346)
- 作者: Thien Tran | +554/-474 | 6 个文件
This a a follow-up of #43273. In this PR, I only focused on optimizing the matrix inverse pipeline inside the KKT kernel. A quick recap on the KKT kernel timeline This is a bit more detailed than the timeline shown previously in #43273, because the extra steps shown above (Prepare gate/beta and Store Ab/Abg to tmem) turn out to be the bottleneck! I asked Codex to add some instrumenting code so we …
- 77654d0 #46777 — [KVTransfer] MultiConnector: merge kv_transfer_params dicts across connectors (#46777)
- 作者: weishu | +9/-6 | 1 个文件
Description: MultiConnector.request_finished previously raised a RuntimeError when more than one child connector returned a non-None kv_transfer_params. This prevented combining connectors that each contribute disjoint keys to the params dictionary—for example, a caching connector returning cached_token_stats alongside a P/D connector returning transfer metadata.
- 75698e6 #47083 — [Bug] Fix sparse attention issue for GLM5.2 non-torch compile path (#47083)
- 作者: Wentao Ye | +4/-4 | 1 个文件
sparse_attn_indexer will allocate a lot of memory during dummy run, we should return after it. CC @WoosukKwon vllm serve zai-org/GLM-5.2-FP8 –kv-cache-dtype fp8_e4m3 –enable-expert-parallel –tensor-parallel-size 4 –tool-call-parser glm47 –enable-auto-tool-choice –reasoning-parser glm45 –port 9256 Originally Now
- 53f7553 #46990 — [ROCm][DeepEP] Stabilize high-throughput DBO for DP+EP (#46990)
- 作者: Andreas Karatzas | +42/-2 | 3 个文件
This fixes ROCm DeepEP high-throughput DBO accuracy for DP+EP without skipping the backend. The failing local repro was test_dbo_dp_ep_gsm8k[deepep_high_throughput], where GSM8K accuracy dropped below the test threshold. DeepEP high-throughput uses Buffer-owned communication state. Under DBO on ROCm, overlapping ubatches can reuse that communication workspace before the previous dispatch/combine h…
- 8fc1b2d #46659 — Fix FA4 dynamic_causal for full attention layers (#46659)
- 作者: Matthew Bonanni | +4/-1 | 1 个文件
Depends on https://github.com/vllm-project/flash-attention/pull/154, merge that first Currently, FA4 doesn’t actually use dynamic causal in full attention layers because the backend hands the per-request causal tensor to the kernel as dynamic_causal but then calls the kernel with causal=False, so it never actually reads the causal tensor. This causes it to run full bidirectional attention for ever…
- 5316638 #47015 — Fix transient dependency issues caused by
requirements/common.txt(#47015)- 作者: Harry Mellor | +638/-72 | 8 个文件
- Fixes the test so that it passes before and after 0.2.3 - Fixes the requirements/test/*.in files so that everything installed during testing is pinned, no more possibility for transient dependency issues
- 61ab70e #42406 — [Model Runner V2] support mamba hybrid models align prefix cache (#42406)
- 作者: zhrrr | +1045/-132 | 9 个文件
A follow PR of https://github.com/vllm-project/vllm/pull/35520 Support mamba_cache_mode=“align” for Mamba prefix caching with Model Runner V2. The implementation keeps the align-mode state handling inside MambaHybridModelState, avoids CPU/GPU synchronization in the hot path. ## Design 1. No post-copy needed: The kernel reads from the src block (previous running block) and writes to the window …
- a309d4f #43729 — Support DCP with FlashInfer MLA (#43729)
- 作者: Woosuk Kwon | +11/-5 | 2 个文件
FlashInfer MLA supports LSE since https://github.com/flashinfer-ai/flashinfer/pull/3116 This PR can be merged once it’s included in GA.
- 72f6399 #46987 — [XPU] [RMSNorm] revert weightless change on xpu (#46987)
- 作者: zofia | +4/-11 | 1 个文件
On XPU, weightless RMSNorm (weight=None) introduced by PR falls back to the native eager implementation, which decomposes into many small aten ops (pow, mean, rsqrt, mul, add). At small batch sizes, where each kernel is tiny and the model is host/dispatch-bound, this fallback significantly increases the number of CPU-dispatched kernels and hurts TPOT.
- 8ad4a01 #46975 — [ModelRunner V2] Simplify recent UnlimitedOCR-related changes (#46975)
- 作者: Nick Hill | +11/-21 | 5 个文件
Recent PR https://github.com/vllm-project/vllm/pull/46564 included some changes to MRV2 that can be simplified.
- 4708292 #46683 — Bump flashinfer version to 0.6.13 (#46683)
- 作者: Wei Zhao | +29/-15 | 6 个文件
Bump flashinfer version to 0.6.13 ## Test Result —
- debec64 #46756 — Add MiniMax-M3 modelopt nvfp4 support (#46756)
- 作者: Jason Li | +93/-9 | 4 个文件
Ports https://github.com/vllm-project/vllm/pull/46380 from minimax-m3-perf onto current main. The original PR was merged into minimax-m3-perf, but current main was still missing the relevant support points when checked locally: ModelOpt mixed MXFP8 dispatch, the parent-prefix fallback for fused projections, and the NVFP4 MoE SwiGLU-OAI alpha/beta/clamp wiring. - Add ModelOpt mixed-precision MXFP8 …
- c8fb296 #46713 — [FS-Offloading] Batch Lookup in C (#46713)
- 作者: Varun Sundar Rabindranath | +162/-2 | 5 个文件
TieredOffloading with FS tier on main yields poor performance. This is mostly due to lookup delays triggered by the FSAsyncLookupManager. We find that the thread backed FSAsyncLookup is severely impacted by the GIL. ### Benchmark This PR: Added unit tests ## Test Result Unit tests pass
- 07d33e5 #44657 — [MyPy] Fix mypy incompatible assignment errors in LRUCacheLoRAModelManager (#44657)
- 作者: Martin Hickey | +7/-39 | 1 个文件
mypy raises two [assignment] errors in LRUCacheLoRAModelManager.init: - vllm/lora/model_manager.py: error: Incompatible types in assignment (expression has type “LoRALRUCache”, base class “LoRAModelManager” defined the type as “dict[int, LoRAModel]”) [assignment] - vllm/lora/model_manager.py: error: Incompatible types in assignment (expression has type “LoRALRUCache”, base class “LoRAModelMana…
- 6185d73 #46827 — [Rust Frontend] Keep literal “null” string for string-typed tool params (#46827)
- 作者: Blas Rodriguez Irizar | +48/-17 | 2 个文件
A tool argument whose schema type is string and whose value is the literal text “null” is coerced to JSON null, so the real string value is lost (e.g. location=“null” becomes null). This affects the XML-style parsers that route through convert_with_optional_schema (qwen_coder, glm_xml, deepseek_dsml, minimax_m2/m3, hy_v3). This diverges from the Python coerce_to_schema_type, which keeps “null” as …
- bc8481a #43373 — [MoE Refactor] Standardize Humming MoE experts + utilities (#43373)
- 作者: bnellnm | +1178/-392 | 8 个文件
Make Humming MoE + Quantization more standard. - Add proper config support check to Humming experts. - Have a single conversion function for preparing a layer for humming MoE kernels convert_to_humming_moe_kernel_format - Make humming experts use the standard apply signature and workspace management. Note: QuantKey conversion utilities generated with claude. cc @jinzhen-lin Tested the mxfp4 quanti…
- 59575da #47008 — [XPU] exclude unsupported models for test_tensor_sechma.py (#47008)
- 作者: Yan Ma | +15/-3 | 2 个文件
Some multi-modality models(mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4, baidu/UnlimitedOCR, etc) are not supported on XPU platform and should excluded in tests like test_tensor_schema.py. pytest -s -v tests/models/multimodal/processing/test_tensor_schema.py ## Test Result —
- a4e3cb4 #47018 — [mypy] Enable mypy for tests directory (#47018)
- 作者: Martin Hickey | +76/-1 | 1 个文件
MyPy was disabled for tests/ because it only checked for files in the vllm/ directory. This commit enables it and adds all relevant directories to the list to be fixed SEPARATE_GROUPS. This is next step of #26533 to enable MyPy for the project. Note: This PR not fix the MyPy errors for the tests/ directory. This will be done in a step by step basis in subsequent PRs.
- ab132ee #46567 — Fix model info cache for package models (#46567)
- 作者: soaringk | +37/-2 | 2 个文件
Fix model-info cache invalidation for package-style model registrations. Some registered model entries point to a package entry point that re-exports classes implemented in submodules. If the cache key only covers the package init.py, changes in those implementation submodules may not invalidate cached model capability metadata. This was found while reviewing the MiniMax-M3 PP work in #45810, …
- 9e86352 #47011 — [CI Failure] Add transformers version check for openai/privacy-filter (#47011)
- 作者: wang.yuqi | +4/-0 | 1 个文件
Ci is currently using transformers==5.5.3, but openai/privacy-filter requires 5.6.0. ## Test Result —
- 5051698 #44589 — Remove unnecessary
load_weightsmethods (#44589)- 作者: Harry Mellor | +821/-1975 | 54 个文件
This PR adds missing functionality to AutoWeightsLoader which allows us to delete the load_weights method boilerplate from 41 architectures in vLLM. Every one of these architectures can automatically load: - GPTQ checkpoints with correct bias skipping - FP8 checkpoints with various scale formats - Checkpoints with fused or sharded qkv_proj/gate_up_proj weights — The specific changes are: - Enabl…
- f6bb868 #47009 — Fix docs on main (#47009)
- 作者: Harry Mellor | +1/-1 | 1 个文件
🐛 Bug Fix
- fcaa84e #47050 — [BugFix] Gate MRV2 mixed sparse-MLA warmup on
max_num_seqs> 1 (#47050)- 作者: Nick Hill | +34/-4 | 3 个文件
Thanks to @ZeldaHuang for finding this
- dc148dc #47157 — [CI][Bugfix] Fix
Hybrid SSM NixlConnector PD prefix cache test (2 GPUs)(#47157)- 作者: Nicolò Lucchesi | +2/-0 | 1 个文件
Fix flaky CI test https://buildkite.com/vllm/ci/builds/74969/list?sid=019f1009-547c-4b3c-9eae-3e500fda953a&tab=output, in which the output of two identical requests might be ~slightly different, causing the assertion to trigger. I currently suspect this might be due to a recent change to FA https://github.com/vllm-project/vllm/pull/36701, which should still be the default on hopper and that would …
- 7cf7cbc #45918 — [Bugfix] MiniCPM-V 4.6: fix grid rows/cols swap in placeholder generation (#45918)
- 作者: tc-mb | +2/-2 | 1 个文件
Fix rows/cols swap in MiniCPM-V 4.6 placeholder generation — grids[0] was read as cols, grids[1] as rows, but all transformers consumers (including HF’s own processor) use the opposite convention. Non-square grids (1×3, 2×3, etc.) produce wrong newline placement and incorrect token counts. Square grids (2×2, 3×3) are unaffected. ## Why only V4.6 V2.0–V4.5 and O series call image_processor.get_slic…
- 00ebf19 #46230 — [Bugfix][Quant] Raise actionable error instead of bare assert for group-size/TP mismatch (#46230) (#46236)
- 作者: Arsalan Shakil | +37/-12 | 6 个文件
- ded6676 #45960 — [Bugfix] Seed RayExecutorV2 TCPStore port by DP rank to avoid collisions (#45960)
- 作者: Seiji Eicher | +64/-1 | 2 个文件
Under data parallel with the Ray V2 executor, each DP engine runs its own RayExecutorV2 and selects the torch.distributed TCPStore port with an independent random get_open_port() in _init_workers_ray. Co-located engines can land on the same port through the pick-release-bind window. A worker then connects to another engine’s store on that port and the handshake value does not match, so distributed…
- aab7af0 #46997 — [Bugfix][ROCm][MLA] Pass q/kv dtypes to get_mla_metadata_v1 in FP8 decode (#46997)
- 作者: peizhang56 | +209/-0 | 3 个文件
Fix a silent accuracy regression for MLA models served with an FP8 KV cache on gfx950 (MI355X) using the AITER MLA backend. With amd-aiter == 0.1.16.post2, Kimi-K2.5 at TP2 (32 q-heads/rank) produces garbage — gsm8k drops from ~0.93 to ~0. ### Root cause AiterMLAMetadataBuilder builds the persistent split/reduce metadata for decode via get_mla_metadata_v1, but did not forward dtype_q /…
- 1907d38 #44002 — [Bugfix] Reject negative values for max_logprobs and long_prefill_token_threshold (#44002)
- 作者: JianweiZheng | +1/-1 | 1 个文件
Two CLI-settable integer config fields silently accept negative values that no validator rejects: - ModelConfig.max_logprobs (vllm/config/model.py:234) — declared int = 20 with no constraint. _validate_logprobs only rewrites the == -1 sentinel; every other negative survives. For logprob-requesting traffic the error message reflects the malformed cap back at the user (“max allowed: -5”); for logpro…
- aed541d #46945 — [Bugfix][Responses] Set completed status for Harmony function calls (#46945)
- 作者: Aman Paswan | +2/-0 | 2 个文件
Fixes #46940. Harmony /v1/responses currently emits completed function_call output items with status: null because _parse_function_call() constructs ResponseFunctionToolCall without setting the status field. This change sets status=“completed” for completed Harmony function calls, matching the existing non-Harmony Responses path and the streaming done-event behavior. ## Test Result - Set status=“c…
- 81bcced #46381 — [Bugfix][ROCm] Preserve MoE weight padding for unquantized Triton path (#46381)
- 作者: Uros Markovic | +7/-0 | 1 个文件
PR #36286 ("[MoE Refactor] Migrate Unquantized to Full Oracle Flow”) introduced a ~8.5% throughput regression for unquantized (BF16) MoE models on ROCm. Root cause. #36286 added an unconditional .contiguous() at the shared return of convert_to_unquantized_kernel_format: On ROCm, _maybe_pad_weight (VLLM_ROCM_MOE_PADDING, on by default) intentionally makes the expert weights non-contiguous —…
- 97b5ce5 #46612 — [Bugfix] Raise VLLMValidationError for non-integer logit_bias keys (#46612)
- 作者: Muhammad Fawaz | +82/-6 | 2 个文件
SamplingParams.from_optional converts logit_bias string keys to int with a bare int() call and no error handling. A non-numeric key (or non-numeric bias value, via the min/max clamp) raises an uncaught ValueError/TypeError. This gets caught by a generic handler further up and returned as a plain 400 with no param field, so the client can’t tell which part of the request was invalid. Part of the er…
- b153dd3 #47074 — [Bugfix] Use larger workspace size for Flashinfer MLA LSE (#47074)
- 作者: Wei Zhao | +18/-9 | 1 个文件
When using Flashinfer MLA LSE with Kimi K2.6, the following error is thrown due to insufficient workspace size: This PR fixes it by using 256 MB workspace size when LSE is used. ## Test Result —
- 930f8dc #46839 — [Bugfix][Rust Frontend] Reject prompt_logprobs for streaming generate (#46839)
- 作者: Reid | +103/-8 | 2 个文件
Fix a validation gap in the Rust /inference/v1/generate route. Before this change, /inference/v1/generate accepted requests with stream=true and sampling_params.prompt_logprobs > 0 or sampling_params.prompt_logprobs == -1. The request returned 200 OK and started an SSE stream, but the streaming generate response shape does not include prompt_logprobs. This change rejects that unsupported parameter…
- fca432e #35076 — [Bugfix] Propagate default stop_token_ids to per-request SamplingParams (#35076)
- 作者: ganesh | +186/-2 | 3 个文件
Fixes #22519 When serving gpt-oss models with tool calling enabled, requests crash with: The root cause is that model-specific stop tokens ( = 200012, = 200002) are correctly loaded into default_sampling_params at server startup, but silently discarded when building per-request SamplingParams. Every other default parameter (temperature, top_p, top_k, min_p, etc.) has fallback logi…
- 5b4cb69 #47079 — [Bugfix][MLA] Fix LSE log-base mismatch in DCP + FlashInfer MLA decode (#47079)
- 作者: Summer Yang | +20/-2 | 3 个文件
FlashInfer’s trtllm-gen MLA decode kernel returns LSE in log2 (per flashinfer’s own reference at flashinfer/trace/templates/attention.py:81: logsumexp / log(2.0)), but MLAAttention’s DCP combine in vllm/model_executor/layers/attention/mla_attention.py hard-codes is_lse_base_on_e=True when calling cp_lse_ag_out_rs / dcp_a2a_lse_reduce, asserting natural log. The mismatch makes the per-shard re-…
- cda05ee #43757 — [Bugfix][Reasoning] Fix thinking_token_budget not enforced on re-entry after forced end (#43757)
- 作者: Ashwin Giridharan | +146/-2 | 2 个文件
Fixes #43708 After ThinkingBudgetStateHolder forces the complete multi-token end string (END_COMPLETE), the state machine fails to detect and enforce budget on subsequent thinking blocks. With ignore_eos=True and max_completion_tokens » thinking_token_budget, the model re-enters
mode unchecked — the parser then returns content=null because it sees without a corresponding …. - c3734e8 #47072 — [CI][Bugfix] Add cohere_melody to ROCm test requirements (#47072)
- 作者: peizhang56 | +3/-0 | 2 个文件
ROCm CI runs pytest -v -s reasoning (.buildkite/test-amd.yaml). tests/reasoning/test_cohere_command_reasoning_parser.py hard-imports the cohere_melody package, which was declared only in requirements/test/cuda.in ~@~T so the ROCm image never installs it and the file errors at collection with ModuleNotFoundError: No module named ‘cohere_melody’. Fix: add cohere_melody to requirements/test/rocm.in a…
- 7be5826 #46986 — [Bugfix] Fix DeepseekV2Model hidden_size (#46986)
- 作者: Jee Jee Li | +4/-4 | 1 个文件
See: https://github.com/vllm-project/vllm/pull/46983 ## Test Result —
- 379acd4 #46860 — [Bugfix][Quantization] Fix W8A8 int-quantized scheme selection regression (#46860)
- 作者: HDCharles | +197/-1 | 2 个文件
(is_static_int8_in and is_static_int8_out) to (is_static_int8_in or is_static_int8_out) accidentally this meant that static_int8_in without static_int8_out would go into the wrong pathway and cause issues. I added unit tests to verify the configurations in general ## Testing vllm serve nm-testing/w8a8_static_asym-e2e (working) Added comprehensive parametrized test test_scheme_selection with 13 tes…
- 36bbecd #46958 — [BugFix] Revert “[KV Offload] Use background thread for mmap / cpu_tensors pinning” (#46958)
- 作者: Varun Sundar Rabindranath | +38/-89 | 1 个文件
Fixes https://github.com/vllm-project/vllm/issues/46933 ## Test Result
- 0ca39c4 #46973 — [Bugfix] Capture final-layer aux hidden state in deepseek_v2 backbone (#46973)
- 作者: Michael Goin | +3/-0 | 1 个文件
The deepseek_v2 backbone captures EAGLE3 / extract_hidden_states aux hidden states only at decoder-layer inputs (ids 0..num_hidden_layers-1), so an aux id equal to num_hidden_layers — the final layer’s output — is silently dropped, returning one fewer hidden state than configured. This makes deepseek_v2-family models (DeepSeek V2/V3, GLM-MoE, Kimi, …) unusable as aux-hidden-state targets wheneve…
- e186107 #45961 — [Bugfix] Use native SiLU activation in CPU fused MoE (#45961)
- 作者: Alden Lobo | +1/-1 | 1 个文件
This PR fixes a startup failure in the CPU fused MoE path for certain Mixture-of-Experts (MoE) models. While starting the vLLM inference engine (version 0.22.0) on a CPU-only system, the initialization sequence can fail with an AssertionError during the warmup/profiling stage. ### Root Cause During the warmup phase on a CPU-only system, the current CPU fused MoE implementation triggers the SiLU pa…
- 0e207da #46835 — [Bugfix] Transformers backend: apply learned lm_head.bias for tied-embedding models (#46835)
- 作者: John | +26/-15 | 3 个文件
The bug vLLM’s Transformers backend (CausalMixin, vllm/model_executor/models/transformers/causal.py) silently drops a learned lm_head.bias for tied-embedding models, producing garbage greedy output. Three independent failure points, all reached when tie_word_embeddings=True: - (A) self.skip_prefixes.append(“lm_head.”) — AutoWeightsLoader._can_skip matches by startswith, so this drops lm_hea…
🔧 Refactor
- 9e84ec8 #46842 — [Refactor] Remove dead minimax allreduce rms kernel (#46842)
- 作者: Wentao Ye | +0/-52 | 4 个文件
We are using minimax_allreduce_rms_qk instead.
🧪 CI/Tests
- ea9ddf5 #45977 — [XPU][CI] Enable shared loader test (#45977)
- 作者: Chaojun Zhang | +28/-1 | 2 个文件
This PR improves stability of test_sharded_state_loader on XPU by forcing max_num_seqs=1 in the XPU test path. - Apply max_num_seqs=1 only when running on XPU. - Keep behavior unchanged for non-XPU platforms.
- ba22cb6 #47000 — [ROCm][Ray][CI] Keep assigned GPU visible for weight transfer (#47000)
- 作者: Andreas Karatzas | +59/-14 | 1 个文件
This fixes ROCm Ray weight-transfer paths by keeping the full GPU topology visible to Ray actors. The weight-transfer tests creates Ray workers that use NCCL/RCCL. On ROCm, Ray may narrow HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES, while vLLM also reasons through CUDA-compatible device APIs. If one actor has a narrowed or mismatched view, RCCL can initialize on the wrong logical device and fail with…
- 4236514 #47004 — [ROCm][CI][Multimodal] Use ROCm-aware FA availability check for Unlimited-OCR (#47004)
- 作者: Andreas Karatzas | +1/-1 | 1 个文件
Use the attention backend helper for the Unlimited-OCR FlashAttention 4 availability check. Unlimited-OCR defaults to FlashAttention 4 when it is available, otherwise it falls back to FlexAttention. The config check imported is_fa_version_supported directly from vllm.vllm_flash_attn, which is CUDA-only in this ROCm environment. On ROCm that import can fail before the fallback decision is reached, …
- 9fc0c08 #47085 — [ROCm][CI] Make tests/v1/shutdown an importable package (#47085)
- 作者: peizhang56 | +0/-0 | 1 个文件
The “Distributed Torchrun + Shutdown Tests (2 GPUs)” step fails on ROCm at tests/v1/shutdown/test_delete.py::test_llm_delete_inprocess with an opaque RuntimeError: Test subprocess ’test_llm_delete_inprocess’ failed (exit code 1), whose real cause is ModuleNotFoundError: No module named ’test_delete’. test_llm_delete_inprocess is wrapped by create_new_process_for_each_test, which resolves to spawn_…
- f2b5fab #47094 — [ROCm][CI] Move LM Eval Large Models (8 GPUs) to mi300 pool (#47094)
- 作者: peizhang56 | +21/-21 | 1 个文件
This moves the “ROCm LM Eval Large Models (8 GPUs)” Buildkite step from the mi325 pool (amdmi325 / mi325_8) to the mi300 pool (amdmi300 / mi300_8), grouping it with the other mi300 LM-eval steps. The test command, source-file dependencies, timeout, and num_gpus are unchanged — only the hardware target moves. This routes the 8-GPU LM-eval job to mi300 8-GPU capacity. AI assistance (Claude) was used…
- 8632c88 #47003 — [ROCm][CI] Use spawn around the threaded OTLP test (#47003)
- 作者: Andreas Karatzas | +70/-51 | 1 个文件
This stabilizes the tracing test on ROCm by avoiding fork-after-gRPC-thread startup and by explicitly cleaning up the engine after the assertion. The test starts a fake OTLP/gRPC server before constructing an LLM. That means the parent process already has gRPC worker threads alive when vLLM launches engine workers. On ROCm CI, that fork pattern can segfault during startup or teardown. The behavior…
- 4eb2279 #45490 — [ROCm][CI] Make memory sampling less racy in tests and sleep mode (#45490)
- 作者: Andreas Karatzas | +59/-7 | 2 个文件
ROCm memory accounting can lag briefly after memory is released, and we were sampling immediately. The affected jobs were: Buildkite: and: Buildkite: For sleep mode, the failing log showed the allocator freeing a large amount of memory, followed immediately by: This is a timing flake. The assertion only fails if the immediate post-sleep mem_get_info() sample is taken before ROCm’s device-wide acco…
- ebcf511 #47067 — [ROCm][CI] Soft Fail
Spec Decode Ngram + SuffixandEntrypoints Integration (LLM)AMD Mirrors (#47067)- 作者: Micah Williamson | +4/-0 | 2 个文件
Entrypoints Integration (LLM) failing: https://buildkite.com/vllm/ci/builds/75108/canvas?jid=019f1368-87e7-4e0f-997b-0783ac50ef9a&tab=output Spec Decode Ngram + Suffix failing: https://buildkite.com/vllm/ci/builds/75164/summary?jid=019f143c-ef4c-4afd-bb36-52077e618271&tab=output#L605
- db28ae2 #46999 — [ROCm][CI] Explicitly tear down multimodal offline LLMs (#46999)
- 作者: Andreas Karatzas | +91/-29 | 6 个文件
AMD CI hit a failure where test_chat_multi_image passed, but the next multimodal LLaVA test failed during engine startup because only about 15 GiB was free on a 192 GiB GPU. That points at the previous test not finishing teardown before the next engine starts. These tests construct large offline LLM instances directly, but teardown was relying on del llm and the global cleanup fixture. The old wea…
🖥️ Kernel
- 8e9d70f #45140 — [Kernel][XPU] Adjust kernel unit tests for XPU (#45140)
- 作者: Agata Dobrzyniewicz | +5/-5 | 1 个文件
Adjustments to kernel unit tests so they pass on Intel XPU. More may be added as they come up. ## Adjustments tests/kernels/mamba/test_mamba_ssm.py — widen bf16 atol for XPU. The selective_state_update Triton kernels accumulate internally in fp32, so on rare elements they are ~1 ULP more accurate than the pure-PyTorch bf16 reference (which accumulates in bf16). With the default bf16 toleranc…
- 6149187 #46819 — [Kernel] Triton MLA logits workspace (#46819)
- 作者: Nicolò Lucchesi | +64/-29 | 1 个文件
TritonMLAImpl.forward_mqa allocates a fresh float32 scratch tensor attn_logits on every decode forward call. This PR addresses the TODO https://github.com/vllm-project/vllm/blob/bf292b5f6b537d154fc09a3b232f89cbc66827f5/vllm/v1/attention/backends/mla/triton_mla.py#L185-L196 by allocating a logits workspace ahead of time, using a single buffer that layer can share. This makes it so that we don’t pay…
- 49e28e8 #44010 — [Kernel][Helion][1/N] Add Helion kernel for fused_qk_norm_rope (#44010)
- 作者: Xiaohong (Sean) Chen | +5911/-0 | 4 个文件
This PR is to add Helion kernel for operation. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: and Autotuning Setup Default “full” autotuning effort. i.e. LFBOTreeSearch with initial_population=FROM_RANDOM, copies=5, max_gen…
🔩 Misc
🦀 Rust Frontend
- 2bc20e8 #46610 — [Frontend] Add Streaming Parser Engine and new Kimi k2.5/k2.6/k2.7 Parser (#46610)
- 作者: Chauncey | +397/-570 | 6 个文件
[Frontend] Add Streaming Parser Engine and new Kimi k2.5/k2.6/k2.7 Parser ## Test Result —
- 3483240 #44512 — [Frontend] Consolidate scale out entrypoints (#44512)
- 作者: wang.yuqi | +422/-289 | 45 个文件
Following #22817 #41907 #44285 Consolidate scale out entrypoints - Tokens IN <> Tokens OUT APIs - Renderer APIs - Derenderer APIs 1. The name “Disaggregated Everything” (#22817) is great, but it is often confused with Disaggregated P/D (Prefill/Decoding) Inference. 2. split ServingRender & ServingDerender keep ci green ## Test Result —