共 87 个 commit，涉及 452 个文件，+16117/-2084 行变动。

概要

统计项 数值
Commit 数 87
变更文件 452
新增行数 +16117
删除行数 -2084

Commit 列表

🐛 Bug Fix

  • 392d1b4 #47725 — [BugFix][LoRA] Refresh punica metadata when LoRA slots are reassigned under an unchanged mapping (#47725)
    • 作者: Ameen Patel | +56/-1 | 2 个文件

    FIX #47724 LoRAModelManager.set_adapter_mapping memoizes on self._last_mapping (the per-token LoRA ID mapping of the running batch) and skips the punica metadata refresh when it is unchanged. The memo is never invalidated when a GPU slot is reassigned: an out-of-band add_lora() on a live engine eagerly activates the new adapter (LRUCacheWorkerLoRAManager.add_adapter -> activate_adapter), which can…

  • 65dcde1 #47466 — [Bugfix] Fix PD disagg + MTP correctness for Qwen3.5(GDN) (#47466)
    • 作者: Dakai An | +22/-22 | 6 个文件

    Problem Qwen3.5-0.8B with MTP=1 can lose GSM8K accuracy under PD disaggregation when concurrency is greater than 1 and default cudagraph is enabled. The issue was reproduced with both Mooncake and NIXL. Trigger pattern: - model: Qwen3.5-0.8B - hybrid/GDN state model - PD disaggregation - MTP/spec decode enabled, num_speculative_tokens=1 - TP=1 - concurrent requests, so decode can batch normal r…

  • 8b74552 #43161 — [Bugfix] Fix UBatchWrapper CUDA graph key to sum all ubatches, not just first two (#43161)
    • 作者: Lanze Liu | +3/-3 | 1 个文件

    When ubatch_size > 2, _capture_ubatches() stored the captured graph under ubatch_metadata[0].num_tokens + ubatch_metadata[1].num_tokens (first two ubatches only), but call looked it up with sum(ubatch_slice.num_tokens for ubatch_slice in ubatch_slices) (all ubatches). The key mismatch caused the cached graph to never be found, triggering unnecessary re-captures on every call. Fixes #43145 Fix …

  • 8b91cd5 #44726 — [Bugfix][Core] Close underlying iterator in merge_async_iterators single-iterator fast path (#44726)
    • 作者: Ting SUN | +31/-2 | 2 个文件

    merge_async_iterators has a fast path for the single-iterator case that yields directly, without the try/finally cleanup the multi-iterator path uses: When the merged generator is closed (a cancelled or disconnected consumer), the underlying generator is never closed and keeps running. The multi-iterator path already aclose()s its iterators in a finally; the single-iterator path does not. For a si…

  • 0a2965b #47318 — [BugFix] Fix ModelOpt mixed-precision quantization for sparse quantized_layers configs. (#47318)
    • 作者: danielafrimi | +66/-10 | 2 个文件

    Fix ModelOpt mixed-precision dispatch so missing sibling layers do not inherit quantization from unrelated siblings. The previous fallback could see one quantized layer under a parent, e.g. o_proj, and incorrectly apply the same quantization to a missing sibling like qkv_proj. This is unsafe for sparse mixed FP8/NVFP4 configs. Now we only infer quantization for known packed projections: qkv_proj <…

  • ed051fa #45418 — [Bugfix] Reject sampling params unsupported by diffusion models (#45418)
    • 作者: Guan-Ming Chiu | +78/-0 | 5 个文件

    Diffusion models ignore per-request sampling params Related: #45163 pytest -v tests/test_sampling_params.py ## Test Result 12 passed —

  • 3354dba #46972 — [Bugfix][KV offload] Store interior chunk-boundary blocks under MTP/Eagle (#46972)
    • 作者: drakosha | +131/-27 | 2 个文件

    Fixes #46971. With MTP/Eagle speculative decoding, OffloadingConnector stored KV to CPU but prefix reuse almost never hit: re-sending an evicted long prompt recomputed the whole prefill instead of loading it from CPU, and External prefix cache hit rate stayed at 0%. _build_store_jobs computes num_blocks twice. The key-collection loop applies the Eagle trailing-block exclusion (num_blocks - 1), but…

  • c5b6623 #47464 — [Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models (#47464)
    • 作者: Kyung Sub Lee (Daniel) | +37/-0 | 2 个文件

    Fix a deterministic engine-core crash for diffusion models (model_config.is_diffusion) caused by the scheduler’s uniform spec-decode padding. The waiting-path padding promotes any request needing exactly 1 token to 1 + num_spec_tokens so decode batches stay uniform for full CUDA graphs. For AR speculative decoding this is harmless — the placeholder drafts simply fail verification. For diffusion mo…

  • ba50b97 #47586 — [Bugfix] Match the mapped filename in find_loaded_library (#47586)
    • 作者: Gabriel Wu | +17/-17 | 1 个文件

    Fix a nondeterministic library hijack in find_loaded_library(). The function picks the first /proc/self/maps line containing lib_name as a plain substring. Two problems compound: 1. Any co-loaded library whose filename merely contains the target matches — e.g. TileLang dlopens its bundled libcudart_stub.so (a linking stub exporting only ~39 cudart symbols) at runtime, and libcudart_stub both…

  • b4cfbc2 #44490 — [Bugfix][Core] Fix host memory leak from undrained new_block_ids (#44490)
    • 作者: Ting SUN | +18/-15 | 3 个文件

    Fixes #44175 Under sustained max_tokens=1 load, host RSS grows linearly with no plateau. The reporter saw ~6.0 → 6.57 GiB over a 12h run with Gemma-3-1b and prefix caching. Who is affected: any V1 model with a full-attention or MLA KV-cache layer and no Mamba layers — i.e. essentially all standard attention models (Llama, Qwen, GPT/OPT, Gemma-2/3, DeepSeek-MLA). Mamba / hybrid-Mamba models…

  • dd5c299 #47201 — [ROCm][Bugfix] Convert ModelOpt FP8 per-channel weights to e4m3fnuz on MI300/MI325 (#47201)
    • 作者: Micah Williamson | +11/-6 | 2 个文件

    We are seeing a CI failure due to the modelopt tests hardcoding the e4m3fn fp8 dtype rather than utilizing the appropriate format for the current platform: https://buildkite.com/vllm/amd-ci/builds/10123/canvas?tab=output&jid=019f057e-31df-4b48-96de-7692de1cea01#L2754 In this PR, we generalize the process_weights_after_loading modelopt method to convert to weights to e4m3fnuz when applicable (detec…

  • cbe9c40 #45352 — [Bugfix] Forward callable hf_overrides to the draft model config (#45352)
    • 作者: Humphrey | +148/-3 | 3 个文件

    Follow-up to #43621 / #45217, resolving the \TODO: revert once figuring out OOM in CI\ for \EagleMistralLarge3ForCausalLM\ in \tests/models/registry.py\ — the non-mocked test path @juliendenize asked for in https://github.com/vllm-project/vllm/pull/43621#issuecomment (review). Root cause of the CI OOM: the draft \ModelConfig\ built inside \SpeculativeConfig.post_init\ always used \hf_overr…

  • 445321f #47780 — [Bugfix] [Quantization] Fix loading for CT DSV2 (#47780)
    • 作者: Kyle Sayers | +1/-1 | 1 个文件

    Purpose ## * Fix loading for DSV2 model + compressed tensors * The deepseek architectures have always included a lot of hard-coding around fp8 and particular quantization config layouts. This PR does not do things like (support non-fp8 quantization of attention layers, support wkw quantization, etc.), but instead only targets ct support for this arch ## Changes ## * Generalize weight_scale_inv …

  • 5769a73 #47550 — [ROCm][CI][Bugfix] Fix flaky parallel tool-call streaming (test assertion + Mistral/Granite parsers) (#47550)
    • 作者: akii96 | +142/-47 | 6 个文件

    test_parallel_tool_calls is flaky on the AMD entrypoints job (API Server Generate) The root cause is that the test incorrectly assumes each streamed chunk holds exactly one tool-call delta: When one decode step crosses the boundary between two parallel tool calls, a single chunk can legitimately carry two deltas: - the tail of call 0, and - the head of call 1. This is valid per the OpenAI streamin…

  • 482e552 #47276 — [Bugfix][ROCm] Fix memory access fault in AITER MLA backend for DPA+FP8 KV (#47276)
    • 作者: Simon Danielsson | +1/-0 | 1 个文件

    Fix memory access fault in DSv3 when running DPA with FP8 KV. Fix is to cast Q to FP8 whenever KV is FP8 as Q is already quantized to FP8 in this case and the MLA decode kernel expects Q to be FP8 as well. The issue is only observed with DP, not with TP. This is because MLA metadata tiles are sized based on dtypes OR when there are TP8 head shapes. The following snippet from AITER summarizes w…

  • 567a784 #40589 — [Bugfix] Fix dp mtp hang (#40589)
    • 作者: SherryC41 | +45/-18 | 1 个文件

    Add a drafter.dummy_run() when proposed_daft_token_ids is skipped to maintain consistency with the behavior of dummy_run executed by other DP. I runned the DeepSeek-R1 bench test with dp 4 tp 2 mtp 3, and set the max-model-len as 4100 to meet the limit greater than input-len+output-len. But found a hang issue caused by one dp’s proposed_daft_token_ids skipped for max_seq_len(4098) + num_spec_token…

  • 04adc88 #47716 — [Bugfix]Fix DeepSeek-V4 fp8_ds_mla KV cache reshape (#47716)
    • 作者: Zhiyi Hong | +14/-2 | 3 个文件

    Fixes #47648. DeepSeek-V4 FlashMLA uses a custom fp8_ds_mla KV layout with 584 bytes per token. The main MLA and SWA cache specs already carry cache_dtype_str, but they did not consistently expose kv_quant_mode, and the GPU model runner used the global cache dtype when computing backend KV cache shapes. As a result, DSpark serving on H200 could allocate/reshape SWA KV cache as the semantic 512-byt…

  • b136cc2 #45965 — [Bugfix][Model] Add stability window to DiffusionGemma to match HF stability_threshold semantics (#45965)
    • 作者: Nathaniel McVicar | +4/-1 | 1 个文件

    DiffusionGemma’s denoise sampler commits a canvas one step too early, dropping the first answer token on short, high-confidence prompts (e.g. “The capital of France is” returns "" instead of “…Paris.”). The checkpoint sets stability_threshold in the HF convention (StableAndConfidentStoppingCriteria): value k requires k + 1 consecutive identical argmax canvases. _compiled_sample_step currently tr…

  • 24dd2ae #46168 — [Bugfix] Preserve FP8 indexer WK pairs across incremental load_weights (#46168)
    • 作者: lcheng | +4/-1 | 1 个文件

    Fixes #46078. _pending_wk_fp8 was local to each DeepseekV2Model.load_weights() call. When a streaming loader (e.g. S3 / RunAI) delivers indexer.wk.weight and indexer.wk.weight_scale_inv in separate load_weights() calls, the pair never completes inside _try_load_fp8_indexer_wk(), so the fused wk_weights_proj.weight is never dequantized from FP8 and stays non-finite — producing corrupted sparse-MLA …

  • 8d8ec38 #47429 — [Bugfix][Spec Decode] Add missing draft_id_to_target_id to DSparkDeepseekV4ForCausalLM (#47429)
    • 作者: Laurent-Zhang | +2/-0 | 1 个文件

    What this PR does Fixes #47418 DSparkDeepseekV4ForCausalLM is a full-vocab draft model (draft ids are target ids, no d2t remapping). However, DSparkSpeculator.load_draft_model accesses model.draft_id_to_target_id unconditionally, causing an AttributeError when running DeepSeek-V4-Flash with DSpark. ### Root Cause In vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:86: This line assumes the d…

  • 598d511 #47589 — [Bugfix][Distributed] Delegate MNNVL allreduce one-shot selection (#47589)
    • 作者: jesco | +60/-13 | 2 个文件

    Fixes #47284. After #47219, vLLM can default FlashInfer fused allreduce/RMSNorm to the mnnvl workspace backend on single-node setups. The fusion call still forced use_oneshot from vLLM’s legacy per-rank threshold table, which can disagree with FlashInfer MNNVL’s AUTO workspace sizing. When vLLM forces use_oneshot=True for a shape that FlashInfer sized for AUTO/two-shot, FlashInfer reports an insuf…

  • 095adf1 #47671 — [Bugfix] Fix int32 overflow in triton_decode_attention page offsets (#47671)
    • 作者: Yifan Qiao | +2/-2 | 1 个文件

    _fwd_kernel_stage1 / _fwd_grouped_kernel_stage1 load kv_page_number as int32 and multiply it by the KV buffer page stride in int32. With MLA c_kv (576 dims) and PAGE_SIZE 480 the page stride is ~276k, so page numbers above ~7.7k overflow to negative offsets → CUDA illegal memory access. Hybrid models hit this quickly: all layer groups share one block pool, so a 90GB KV buffer already holds ~163k p…

  • 373eb31 #46066 — [Bugfix][Core] Fix num_output_placeholders underflow with async scheduling + spec decode (#46066)
    • 作者: Ting SUN | +58/-3 | 3 个文件

    Under async scheduling with speculative decoding, reset_prefix_cache(reset_running_requests=True) force-preempts the running requests, zeroes each request’s num_output_placeholders, and records the in-flight frame count in async_tokens_to_discard so those now-stale frames are dropped when they return. This path is reached by the /reset_prefix_cache dev endpoint and, more commonly, by RLHF weight u…

  • 7a90eb9 #47091 — [Bugfix] [Gemma4] Fix Gemma4 MTP draft model layers ignoring quant_config (#47091)
    • 作者: Ayushman Singh | +13/-4 | 1 个文件

    This PR fixes quantization support for the Gemma4 Eagle (MTP) draft model in speculative decoding. Previously, all layers in the Gemma4 MTP draft model were hardcoded to quant_config=None, causing quantized draft checkpoints to silently load incorrectly and produce 0% draft token acceptance. This PR addresses the following: - Use get_draft_quant_config in Gemma4MultiTokenPredictor and Gemma4MTP to…

  • 98e4726 #47697 — [fix][run_batch]: respect proxy env vars when downloading media URLs (#47697)
    • 作者: Qiming Zhang | +4/-12 | 1 个文件

    aiohttp.ClientSession does not read HTTP_PROXY/HTTPS_PROXY environment variables by default. This causes download_bytes_from_url to fail with a connection timeout in network environments that require a proxy (e.g., corporate networks), even though tools like curl/wget work fine. Add trust_env=True to aiohttp.ClientSession() in download_bytes_from_url so that aiohttp respects the system proxy setti…

  • 40cc2e8 #47165 — [Bugfix] Return HTTP 422 for unprocessable image URLs instead of 500 (#47165)
    • 作者: Alexis K. | +288/-12 | 5 个文件

    Associated issue Issue 47163 - [Bug]: Image URL errors return HTTP 500 instead of 422 for unprocessable content This PR fixes the issue where vLLM returns HTTP 500 (Internal Server Error) when image URLs in requests point to non-existent or inaccessible resources. The fix ensures these client-side errors properly return HTTP 422 (Unprocessable Entity), making it clear that the issue is with the…

  • 90ce3a0 #47607 — [bugfix] fix MOSS-Audio deepstack_input_embeds initialization in PP (#47607)
    • 作者: Yan Ma | +4/-0 | 1 个文件

    After PR #44443, all dense models use ModelRunnerV2 as default and so does MOSS-Audio. And will run into below errors: verified on L20. pytest -s -v tests/models/multimodal/generation/test_moss_audio.py::test_moss_audio_parallel_smoke[pp2] ## Test Result —

  • 26c754d #47116 — [XPU][Bugfix] Do not transpose weight_scale_inv at load time (#47116)
    • 作者: Ma Jian | +2/-24 | 1 个文件

    The MLA attention’s _o_proj path (via _get_cached_wo_a_bf16 in rocm_aiter_mla_sparse.py) directly reads weight_scale_inv and assumes the original [N/128, K/128] checkpoint layout for manual dequantization. Pre-transposing to [K/128, N/128] in process_weights_after_loading caused _expand_2d_block_scales to apply wrong scale values (row/col blocks swapped), silently degrading GSM8K accuracy from 96….

  • d039c17 #47448 — [Bugfix] Recycle post-final-norm hidden in GLM MTP (single norm) (#47448)
    • 作者: xiaozhoupy | +16/-7 | 1 个文件

    The compile-free DeepseekV32 MTP draft layer returned the pre-final-norm hidden as both the draft-logits hidden and the recycled previous_hidden_states. Recycling the pre-final-norm hidden mismatches the draft model’s hnorm and drops MTP acceptance; the reference deepseek_mtp.py (PR #45895) recycles the post-final-norm hidden. Fix by computing the post-final-norm hidden once – fusing the residual…

📦 Other

  • 21b396a #47784 — AGENTS MD: Add suggestion on how to incorporate tests (#47784)
    • 作者: Simon Mo | +30/-19 | 1 个文件

    I have seeing more PRs with newly created test files by agents.

  • bdaf275 #47868 — [XPU] Fix Event init failure w/ blocking (#47868)
    • 作者: liuzhenwei | +7/-1 | 1 个文件

    Error msg: ## Test Result —

  • beb4327 #47745 — Enable causal masking for SWA in vllm-project/speculators models (#47745)
    • 作者: Eldar Kurtić | +4/-0 | 1 个文件

    Right now, vllm-project/speculators models contain sliding_window_non_causal to indicate that a model was trained with causal masking and SWA. Without this PR, those models are deployed with bidirectional masking, creating a discrepancy between training and inference setup. This mismatch has a negative impact on results, and the PR address the issue. See improved results in the table below: | mode…

  • c46ced1 #46544 — [kv_offload] Establish tier-owned KV event handling (#46544)
    • 作者: Chang Guo | +61/-45 | 11 个文件

    KV event responsibility should follow storage ownership. A tier is the only component that can distinguish an actual store from a no-op and report its own placement changes. Having a parent construct child-tier events would assume a single destination medium and make the take_events() contract harder to reason about. This PR establishes the ownership and aggregation contract. Actual secondary-tier…

  • 65a7b46 #47063 — [KV-Offloading] Support workload identity for objectstore secondary tier (#47063)
    • 作者: Pierangelo Di Pilato | +76/-11 | 3 个文件

    Make access_key and secret_key optional in ObjStoreConfig so that when omitted, the NIXL OBJ plugin falls back to the AWS SDK default credential provider chain (IAM roles, environment variables, credential files, etc.). This enables workload-identity based authentication on Kubernetes (AWS IRSA, GCP Workload Identity, Azure Workload Identity) without requiring explicit credentials in the vLLM conf…

  • 93e2ab7 #45963 — Disable dynamic speculative decoding when DP is enabled (#45963)
    • 作者: Tyler Michael Smith | +60/-1 | 4 个文件

    Disable num_speculative_tokens_per_batch_size when data parallelism is enabled. Dynamic speculative decoding currently picks K from each scheduler’s local scheduled request count. With DP, ranks can choose different K values, and the EAGLE/draft proposer has K-dependent DP collectives. This change warns and falls back to the static num_speculative_tokens value when data_parallel_size > 1. Some cod…

  • 9204699 #38641 — [UX] Log worker exit code when process dies unexpectedly (#38641)
    • 作者: Nick Cao | +5/-2 | 1 个文件

    When a worker process dies, the MultiprocWorkerMonitor logs the process name but not the exit code. This makes it impossible to distinguish between an OOM kill (exit code -9 / SIGKILL), a segfault (-11 / SIGSEGV), a Python exception (1), or other causes without external tools like dmesg. Log proc.exitcode alongside the process name so users and developers can immediately identify the class of fail…

  • dd94484 #41359 — Bump Transformers version to 5.10.4 (#41359)
    • 作者: Harry Mellor | +271/-234 | 51 个文件

    TL;DR which PRs do I need in a patch release of Transformers: - https://github.com/huggingface/transformers/pull/46456 - https://github.com/huggingface/transformers/pull/46500 - https://github.com/huggingface/transformers/pull/46524 - https://github.com/huggingface/transformers/pull/46525 - https://github.com/huggingface/transformers/pull/46605 - https://github.com/huggingface/transformers/pull/46…

  • 7ff656c #47440 — fix: ensure no double load of lm head in nemotron mtp (#47440)
    • 作者: Shaun Kotek | +1/-5 | 1 个文件

    fix nemotron mtp head loading for quantized heads load nemotron 3.5 nano, v3 super and v3 ultra and see that it works. make sure AL is still good and no accuracy change ## Test Result All tests pass. — - [X] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”. - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the …

  • 48fcfc9 #47274 — [KV Offload] Add ParentManager ABC for secondary tier callbacks (#47274)
    • 作者: Ronen Schaffer | +169/-44 | 2 个文件
    • Add ParentManager ABC enabling secondary tiers (e.g. P2P) to call back into the TieringOffloadingManager for block lookup, pinning, and request lifecycle management - Implement _SecondaryTierFacingParent wrapper that auto-injects exclude_tier to prevent fan-out loops - Add serve_external_requests() hook for secondary tiers to process remote requests - Extract create_store_job() from duplicated i…
  • b3e85be #47834 — fix: use configured max_logprobs instead of hardcoded 20 in derender validation (#47834)
    • 作者: Juan Pérez de Algaba | +13/-4 | 2 个文件
    • Replace the hardcoded top_logprobs limit of 20 in _validate_derender_bounds() with self.model_config.max_logprobs, so the derender endpoints respect the server’s –max-logprobs setting. - The default value (20) comes from the OpenAI Chat Completions API spec which defines top_logprobs as “an integer between 0 and 20”. - When max_logprobs is -1 (no cap, via –max-logprobs -1), the check is skippe…
  • c85d720 #47321 — [HARDWARE][POWER] optimize math functions of VSX power (#47321)
    • 作者: Rukhaiya2004 | +154/-54 | 5 个文件

    Adds PowerPC VSX (Vector Scalar Extension) support to the vLLM CPU attention backend for optimized inference on POWER architecture systems.

  • 066f02a #47427 — [MoE] FI autotuning: max bucket = max token count [e.g. DP_size*MNBT] (#47427)
    • 作者: Netanel Haber | +38/-2 | 4 个文件

    vLLM does not pass tune_max_num_tokens to FlashInfer’s BF16, FP8, and NVFP4 TRT-LLM MoE kernels. FlashInfer therefore ends autotuning at its default maximum bucket of 8192. This can be too low. The estimated maximum token count is: DP size * max_num_batched_tokens Larger inputs still run, but use a tactic tuned for a smaller bucket. This PR passes that estimate to FlashInfer while keeping 8192 as …

  • 1e823dc #47830 — [docs update] Update usage of hf cli for cache list and removal (#47830)
    • 作者: Aritra Roy Gosthipaty | +7/-33 | 1 个文件

    This PR updates the docs with the correct (updated) usage of the HF CLI for cache list and removal. CC: @hmellor

  • 8e61b64 #47260 — fix(security): add resource bounds validation to derender endpoints (#47260)
    • 作者: Juan Pérez de Algaba | +272/-3 | 3 个文件

    The /v1/completions/derender and /v1/chat/completions/derender endpoints accepted caller-supplied GenerateResponse objects with unbounded token_ids, choices, generate_responses, and logprob structures, processing them via tokenizer.decode() without enforcing the resource limits that the normal generation path applies (CWE-400 / CWE-770). Add _validate_derender_bounds() to ServingDerender that reje…

  • e040899 #45958 — [KV Offloading] Add basic offloading metrics (#45958)
    • 作者: Srinivas Krovvidi | +320/-16 | 10 个文件
    • add vllm:kv_offload_lookup_delay_seconds - add vllm:kv_offload_allocation_failure - add vllm:kv_offload_cpu_allocation_size - add focused scheduler and CPU manager coverage for the new metrics
  • 2f71b2b #47685 — [ROCm] Align mixed encoder-decoder KV cache views in V2 runner (#47685)
    • 作者: Andreas Karatzas | +82/-0 | 1 个文件

    Fixes regression in encoder-decoder models that share one KV cache allocation across attention backends with different physical layouts, introduced by: - #47035 That PR normalized the legacy model runner path, but the failing Buildkite job uses the V2 model runner. In the V2 path, attn_utils._reshape_kv_cache still left shared decoder/cross-attention allocations with mixed layout semantics: decode…

  • 32ab064 #47148 — [UX] Add model_class_overrides for development and debugging (#47148)
    • 作者: Jee Jee Li | +70/-0 | 3 个文件

    We are currently refactoring the model implementation, and may need to switch back and forth between different model paths (e.g., GLM52). To facilitate development and debugging, we are adding model_class_overrides to specify the model’s registration path, avoiding the need to explicitly modify the registry.py, like: ### offline mode ### online mode cc @WoosukKwon ## Test Result —

  • 34e6dfc #47787 — [Rust Frontend] Stamp arrival_time at the frontend entry (#47787)
    • 作者: Tahsin Tunan | +63/-12 | 10 个文件

    Summary Rust stamps arrival_time after render and tokenize. Python stamps it at the renderer entry, before both. So Rust’s TTFT and e2e histograms run short. This stamps it at the frontend entry to match. No engine or protocol change. Refs: roadmap #44280. Part of RFC #44757.

  • 39a1d32 #47581 — [Rust Frontend] Avoid extra copies for multimodal tensors (#47581)
    • 作者: Reid | +67/-4 | 2 个文件

    Multimodal requests with inline tensor payloads can carry large byte buffers, such as image pixel_values. The Rust frontend previously copied those bytes in two avoidable places before sending the request to engine: 1. lower_text_request cloned request.mm_features into GenerateRequest. 2. WireArrayData::RawView cloned the raw tensor bytes while building an owned MessagePack ext value. The change r…

  • 700e882 #46609 — Add TorchCodec as a video decoding backend (#46609)
    • 作者: Nicolas Hug | +300/-12 | 9 个文件

    This PR adds TorchCodec as a video decoding backend, along with the existing opencv and pyav decoders. Like pyav, TorchCodec releases the GIL. ### Things that TorchCodec can do - Simple decoding APIs: just call VideoDecoder(bytes, …).get_frames_at_indices(indices) and you’re done. No need for your own decoding loop like for pyav or opencv - Efficient on sparse sampling: both the opencv a…

  • a4f019f #45159 — fix(distributed): propagate distributed_timeout_seconds to NCCL device groups (#45159)
    • 作者: jialoop-git | +36/-3 | 2 个文件

    Fix #45078: –distributed-timeout-seconds was not being propagated to the NCCL device groups created in GroupCoordinator.init() and make_sibling_device_group(). The timeout was correctly applied to the WORLD group and Gloo CPU groups, but the NCCL device group used for actual GPU collective operations always used PyTorch’s default 600-second timeout, ignoring the user-specified value. This cau…

  • 9dd2465 #35059 — feat(cpu): add CPU support for Mamba ShortConv (#35059)
    • 作者: Rahul Vishwakarma | +277/-4 | 4 个文件

    Adds native CPU execution for Mamba’s ShortConv layer. On CPU, ShortConv.forward_native was a no-op stub (return), so LiquidAI/LFM2-style models produced no conv output. This implements forward_native using the existing CPU causal_conv1d reference ops and routes CPU execution to it. Fixes: #25771

  • a46c932 #47619 — [Rust Frontend] Add DeepSeek V3.2 roundtrip fixture (#47619)
    • 作者: Reid | +14/-0 | 1 个文件

    Adds a DeepSeek V3.2 roundtrip case for the Rust chat frontend. The new case uses deepseek-ai/DeepSeek-V3.2-Exp with the existing DeepSeek V3.2 renderer and auto-selected parser, exercising the DSML tool-call path through render, stream parse, and rerender. ## Test Result —

  • 69f3150 #47253 — [XPU] Fix PP accuracy on XPU device (#47253)
    • 作者: YiSheng5 | +5/-0 | 1 个文件

    Fix the accuracy issue on XPU device.

  • d891b9b #41652 — [Quantization] add humming moe backend to all dense/moe oracles (#41652)
    • 作者: Jinzhen Lin | +1336/-122 | 73 个文件

    This PR add humming moe backend to all dense/moe oracles, and fixed several minor bugs discovered during model testing. Tested with - Dense - compressed-tensors - int4: RedHatAI/Qwen3.5-4B-quantized.w4a16 - int8: nm-testing/Qwen2.5-VL-3B-Instruct-quantized.w8a8 - fp8: - channelwise: nm-testing/Qwen2-1.5B-Instruct-FP8W8 - tensorwise: nm-testing/Llama-3.1-8B-Instruct-KV-FP8-Tensor - blockwise: n…

  • b1384f5 #43328 — Enable B12x backend for non-gated MoEs (like Nemotron) (#43328)
    • 作者: Andrii Skliar | +225/-40 | 3 个文件

    Stacked on top of #40082. This PR refines the FlashInfer B12x MoE integration by switching the SM12x MoE path to FlashInfer’s B12xMoEWrapper API and adding ReLU2 / non-gated MoE coverage. Key changes: - Use B12xMoEWrapper for FlashInferB12xExperts - Keep BF16 hidden states as unquantized inputs; B12x handles FP4 activation quantization internally - Support both SiLU gated MoE and ReLU2 non-gated M…

  • 3ee9eea #47457 — [macOS][CPU][Installation] Fix the broken installation of vllm 0.24.0 in macos + cpu (#47457)
    • 作者: Ranran | +1/-1 | 1 个文件

    Problem vllm-metal installation is broken, because it cannot compile the vllm dependency on mac before this PR. tokenspeed-mla==0.1.2 in requirements/cuda.txt has no platform marker, but the package itself only publishes manylinux wheels (no sdist). Since setup.py’s get_requirements() copies this file’s lines verbatim into vLLM’s published install_requires, the unconditional requirement ships i…

  • 5bce653 #47187 — Make the Transformers modeling backend as fast as native vLLM (#47187)
    • 作者: Harry Mellor | +2679/-160 | 23 个文件

    For the Transformers modelling backend this PR: - enables the use of MergedColumnParallelLinear and QKVParallelLinear - fully leverages MoERunner to enable: - async shared experts - DBO - Sequence parallel - tp_plan becomes optional because the fusion passes know correct the TP orientations of the resulting fusions - pp_plan becomes optional because vLLM now checks for the nn.ModuleList in the tex…

  • 07f9baf #47140 — Revert “[Platform] Replace torch.cuda.Event with torch.Event (#47140)” (#47668)
    • 作者: Kunshang Ji | +70/-60 | 31 个文件

    part of https://github.com/vllm-project/vllm/issues/30679 CI ## Test Result —

  • 8f4c69b #47444 — [Rust Frontend] Cache metric handles for scheduler & request stats (#47444)
    • 作者: Bugen Zhao | +343/-237 | 9 个文件

    This PR reduces repeated get_or_create lookups on scheduler & request-level metrics paths. Instead of calling get_or_create every time when we want to access the global metrics handle for a request, we pre-resolve per-engine metrics handles once ClientInner and GenerateOutputStream are created, and cache them for future use. Still keep dynamic-label metrics, such as spec decoding accepted tokens p…

  • 8b79971 #43597 — attention: pass None for unused args in unified attention TD path (#43597)
    • 作者: Artur Fierka | +29/-20 | 1 个文件

    Performance follow-up to #40327 (UA+TD). On XPU we measured +1-10% throughput in some configurations. Co-authored with: @quinnlp

  • 740f379 #46065 — [ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator (#46065)
    • 作者: BadrBasowid | +499/-148 | 12 个文件

    Previously, with the AITER fused AR-RMSNorm enabled, two separate custom-allreduce instances were live: vLLM’s CustomAllreduce for the plain allreduce, and a standalone AITER CustomAllreduce (rocm_aiter_ops._CUSTOM_ALL_REDUCE) for the fused op, each with its own IPC buffers, both registered during CUDA graph capture. Performance was the main reason it’s done this way rather than by dropping the AI…

  • ba22152 #47259 — fix(security): block request-level GPU video backend selection withou… (#47259)
    • 作者: Juan Pérez de Algaba | +131/-8 | 4 个文件

    Prevent request-level media_io_kwargs from selecting GPU video backends (e.g. pynvvideocodec) when the engine did not reserve VRAM for them at startup. VideoMediaIO.merge_kwargs() now strips GPU backend overrides from runtime kwargs unless the static config already named that backend, closing a resource exhaustion vector where unreserved CUDA context and decoder-surface allocations could starve th…

  • 736f1a5 #47688 — [XPU] Route mm_prefix models to Triton attention backend (#47688)
    • 作者: liuzhenwei | +9/-0 | 1 个文件

    mm_prefix (prefix-LM bidirectional mask) is only supported by FA4, which is unavailable on XPU, so fall back to Triton attention. pytest -s -v tests/models/multimodal/generation/test_mm_prefix_lm.py::test_mm_prefix_lm_e2e ## Test Result —

  • 344609a #47695 — [CI/Build] Fix pre-commit check (#47695)
    • 作者: Li, Jiang | +7/-8 | 1 个文件

    Test Result —

  • 2fa1056 #47420 — [Core][DP] Rotate load-balancer tie-break to avoid systematic engine bias (#47420)
    • 作者: Qiming Zhang | +6/-0 | 1 个文件

    Motivation In DPLBAsyncMPClient.get_core_engine_for_request, the internal DP load balancer selects the engine with the lowest score (waiting * 4 + running), scanning from a fixed eng_start_index and using a strict < comparison. When engines are tied, the winner is always the first one scanned — i.e. eng_start_index, which is constant for a single front-end client (0). Between coordinator stats…

🧪 CI/Tests

  • 0ed05b6 #47832 — [CI] Fix Transformers modeling backend LoRA test (#47832)
    • 作者: Harry Mellor | +8/-18 | 3 个文件

    As identified in https://github.com/vllm-project/vllm/pull/47804, https://github.com/vllm-project/vllm/pull/47187 added a new attribute to QKVParallelLinear for the split operation that is added as part of Transformers backend fusion. This new attribute was not copied over by LoRA wrapping classes which caused a test failure. This PR removes the need for this new attribute as all the necessary inf…

  • cbb5f04 #46904 — [ROCm][CI] Refresh ROCm base images when docker rocm_base changes (#46904)
    • 作者: Andreas Karatzas | +724/-52 | 14 个文件

    ROCm Dockerfile.rocm_base changes currently do not have a clean release path through CI. A PR can change the base image recipe while the rest of the ROCm image stack is still validated against older base/ci_base images. This makes base-image changes harder to reason about and easy to under-test. This PR adds an explicit base-refresh lane to the AMD image build: - Check whether docker/Dockerfile.ro…

  • e55cc59 #47735 — [Rust Frontend][CI] Unblock more end-to-end test cases (#47735)
    • 作者: Bugen Zhao | +105/-40 | 14 个文件

    Unblock some more end-to-end test cases in CI for Rust frontend as we’re filling more gaps. This PR only enables the test cases that require no or minimal code and test changes. For those requiring a larger feature scope or fixes, separate follow-up PRs will be opened to unblock them. ## Test Result —

  • 6db31c8 #47758 — [XPU][CI]Adjust memory request for tests in Intel GPU CI (#47758)
    • 作者: xiangdong | +9/-9 | 5 个文件

    Adjust memory request for tests in Intel GPU CI ## Test Result —

  • 8484ca5 #47478 — [ROCm][CI] Adding Rust parity (#47478)
    • 作者: Andreas Karatzas | +153/-1 | 3 个文件

    Adds ROCm AMD CI coverage for Rust frontend cargo style, clippy, and tests on the MI250 pool. It also adds MI300 Rust frontend parity jobs for OpenAI coverage, serve admin coverage, core correctness, tool use, and distributed paths. Dockerfile.rocm now includes the Rust sources, toolchain file, and unzip dependency needed for these jobs. The cargo CI helper can also resolve the repo root when the …

  • ae098ab #47726 — [CI] Fix some errors on main (#47726)
    • 作者: Harry Mellor | +39/-22 | 2 个文件
  • 51ee564 #47748 — [CI] Skip test for checkpoint that was deleted (#47748)
    • 作者: Harry Mellor | +1/-0 | 1 个文件

    Tencent just deleted tencent/HunyuanOCR and most of vLLM’s CI runners have broken HF caches. This PR marks the checkpoint as not available online so that it deosn’t block the rest of CI.

  • f676808 #47730 — [CI] Use TTY for AMD CI tests for colored buildkite logs (#47730)
    • 作者: Nick Hill | +4/-1 | 2 个文件

    For other tests we launch the docker container with a tty, unless there’s a reason not to I’m unaware of it would be good to do so for the AMD tests too for consistency. This also means we’ll get nicer logs in builkite with the ansi colors.

  • cdab283 #47675 — [XPU][CI]Add agent tags for Basic Models Tests (Initialization) in Intel GPU CI (#47675)
    • 作者: xiangdong | +4/-0 | 1 个文件

    Test Result —

  • fb265fc #47591 — [ROCm][CI] Increasing parallelism in Basic Models Tests (Extra Initialization) (#47591)
    • 作者: Andreas Karatzas | +6/-6 | 1 个文件

    Increasing parallelism in Basic Models Tests (Extra Initialization) %N to avoid timeouts like: - https://buildkite.com/vllm/amd-ci/builds/10424/list?jid=019f29a6-adae-4dae-9bbe-dd8444e163df&tab=output

  • 8f0e75e #47481 — [ROCm][CI] Adding nixl multiconn (#47481)
    • 作者: Andreas Karatzas | +88/-7 | 4 个文件

    Adds MI300 ROCm CI jobs for Hybrid SSM NixlConnector prefix cache coverage and MultiConnector NIXL plus offloading accuracy and edge case coverage. The new jobs run on the 2-GPU MI300 pool and use the TRITON_ATTN backend. The NIXL integration scripts now accept ATTENTION_BACKEND and VLLM_SERVE_EXTRA_ARGS so the same scripts can be reused across backend-specific ROCm coverage. The MultiConnector sc…

⚡ Performance

  • d3e69fd #47081 — [Perf] Use blocking CUDA events to avoid busy polling cuda driver lock (#47081)
    • 作者: Summer Yang | +13/-6 | 3 个文件

    When async scheduling is enabled, the per-step CUDA-event syncs default to spin-wait events that busy-poll the CUDA driver lock. Under per-step TP collective contention the main-thread spin can balloon on a rotating rank, making it arrive late at the eager cross_device_reduce all-reduce while the other ranks spin-wait on it — the occasional “long all-reduce” straggle. This makes those events **b…

  • c64c356 #45672 — [Perf] Bound DiffusionGemma sampler transient via request-tiled logits (#45672)
    • 作者: Guan-Ming Chiu | +104/-90 | 1 个文件

    Bound the DiffusionGemma sampler’s transient logit memory. It scales with max_num_seqs and OOMs before weights or the KV pool do. ## Implementation - Tile _compiled_sample_step over request groups; the group size is auto-sized from free GPU memory (mem_get_info), no tuning knob. Tiling is bit-identical to a single pass. - Compute logprobs per tile so the fp32 scaled-logits buffer frees each iterat…

  • 5ad1117 #47416 — [perf]Add fused Kimi image preprocessing (#47416)
    • 作者: Kevin_Xiong | +402/-2 | 5 个文件

    This PR adds a fused CPU image preprocessing path for Kimi-K2.5/K2.6 vision chunks. - Adds KimiK25FusedVisionProcessor, a vLLM-local processor that preserves the Kimi NaViT resize/token semantics while fusing padding, normalization, and patchification through numba. - Uses the fused processor automatically when numba is available. - Falls back to the remote HF image processor when numba is unavail…

  • f70caef #47474 — [Perf] Cache token_to_req_indices for dsv4, 5x~6x kernel performance improvement (#47474)
    • 作者: Wentao Ye | +34/-25 | 4 个文件

    So we save two additional copy from CPU to GPU ### Acc ### Perf Running this AI-generated microbenchmark Script And we will get 5x~6x kernel level improvement.

🖥️ Kernel

  • 5d23ca4 #47408 — [Kernel] Applies routed_scaling_factor internally (#47408)
    • 作者: Jee Jee Li | +56/-44 | 8 个文件

    Move the routed_scaling_factor for both M3 and DeepSeek V3 into the routing kernel. This has two benefits: - When torch.compile is not enabled(new model path), it eliminates the extra mul kernel introduced by routed_scaling_factor - It makes it easier to support the fusion of finalize + add + ar + norm #### before #### after ## Test Result —

  • 9fde043 #43994 — [Kernel][Helion][1/N] Add Helion kernel for silu_and_mul_per_block_quant (#43994)
    • 作者: Xiaohong (Sean) Chen | +4782/-0 | 4 个文件

    This PR is to add Helion kernel for operation. This is a subtask for https://github.com/vllm-project/vllm/issues/32962. ### Kernel level benchmark Environment Python: 3.12.12 Pytorch: 2.11.0 Cuda: 13.0 Helion: 1.0.0 Benchmark Setup Latency measure: Baseline: and Autotuning Setup Default “full” autotuning effort. i.e. LFBOTreeSearch with initial_population=FROM_RANDOM, copies=5, max_gen…

🦀 Rust Frontend

  • 86db6c3 #46768 — [Frontend] add per-request timing metrics field to response body of Chat/Completions APIs (#46768)
    • 作者: nv-nedelman-1 | +565/-4 | 12 个文件

    Motivation for this feature is found in the RFE: https://github.com/vllm-project/vllm/issues/40076 There are compelling use cases for this feature related to billing, SLAs, experimentation, and debugging. These use cases are described in the RFE discussion/comments and generally require that token/engine level metrics be exposed in the response body. To that end, this PR introduces a metrics top-l…

🔧 Refactor

  • b1c6dba #47329 — [Refactor] Remove multiple dead code (#47329)
    • 作者: Wentao Ye | +0/-186 | 7 个文件

    Remove multiple dead code

📖 Documentation

  • 641cb59 #45813 — [Doc] Clarify fastokens availability (#45813)
    • 作者: LiJzd | +3/-2 | 1 个文件

    Fixes #43446 - Clarifies that VLLM_USE_FASTOKENS is available in vLLM v0.23.0 and later. - Tells users to upgrade vLLM if their installed version does not recognize the environment variable. ## Duplicate-work check - Checked vllm-project/vllm#43446 comments. - Checked open PRs with 43446 in:body: no matches. - Checked open PRs with VLLM_USE_FASTOKENS fastokens: no matches. ## Validation - git diff…

  • 3d7f357 #47701 — [Doc] docs: fix note formatting for pooling models (#47701)
    • 作者: Sungjae Lee | +2/-3 | 1 个文件

    The purpose of this PR is fixing note part of pooling models docs. ## Test Result —