共 97 个 commit，涉及 428 个文件，+17441/-4073 行变动。

概要

统计项	数值
Commit 数	97
变更文件	428
新增行数	+17441
删除行数	-4073

Commit 列表

📦 Other

• e7df232 #46252 — [KV Offload] Gate packed HMA KV cache on cross-layer config (#46252)
  • 作者: Lucas Wilkinson | +42/-33 | 4 个文件

  • Use kv_connector_extra_config[“enable_cross_layers_blocks”] to opt multi-group HMA layouts into packed KV allocation as laid out in https://docs.vllm.ai/en/stable/features/nixl_connector_usage/#cross-layers-blocks - Keep DeepSeek V4-style UniformTypeKVCacheSpecs layouts on the packed path by default. - Remove the extra VLLM_USE_PACKED_HMA_KV_CACHE environment flag. - Canonicalize packed KV cache…

• b3a688c #46548 — [ROCm] Fix OOB During Model Warmup With ROCM_ATTN and MRV2 (#46548)
  • 作者: Micah Williamson | +8/-8 | 3 个文件

  Resolves https://github.com/vllm-project/vllm/issues/46179 (as well as numerous other failing tests in AMD CI, e.g. pytest -v -s tests/compile/fullgraph/test_full_graph.py::test_full_graph[neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic-model_kwargs1-2]). This PR resolves a latent seg fault in the ROCM_ATTN that was exposed when Llama was added to the default architectures for MRV2 in https://github…

• 1cd3e0e #46627 — [Bug] Fix IndentationError: expected an indented block after 'with' statement (#46627)
  • 作者: Wentao Ye | +2/-7 | 1 个文件

  Seems to be introduced by https://github.com/vllm-project/vllm/pull/44483 This PR fixes the issue

• f889325 #45850 — [KV Offload] Use background thread for mmap / cpu_tensors pinning (#45850)
  • 作者: Yiwei Hu | +89/-38 | 1 个文件

  This pull request introduces improvements to how CPU tensors / mmap are pinned for CUDA transfers in vllm/v1/kv_offload/cpu/gpu_worker.py, including asynchronous pinning for better startup performance. The changes add a dedicated function for pinning CPU tensors, enable background pinning on CUDA platforms, and ensure logging and synchronization of the pinning process at the end of CpuGpuOffloadin…

• bb61177 #46363 — [KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363)
  • 作者: Ronen Schaffer | +323/-172 | 14 个文件

  Replace the ambiguous bool | None return type of OffloadingManager.lookup() with a LookupResult enum that distinguishes between two previously conflated None cases: - HIT_PENDING — block exists in the primary tier but its write is still in-flight. Should count as a hit in _sliding_window_lookup’s consecutive-hit streak. - RETRY — block found on a secondary tier (promotion started) or async…

• 007b5a5 #46511 — [Log] Update to log once (#46511)
  • 作者: Wentao Ye | +23/-21 | 4 个文件

  Some infos are not needed to replicate, this PR fixes the issue

• 61ee183 #46414 — [ROCm] Fix AITER FP8 quantization schema tests (#46414)
  • 作者: djramic | +29/-59 | 1 个文件

  The AITER FP8 quant schema tests fail on MI300: - test_per_tensor_quant_torch_compile errors with error_on_custom_op_aliasing does not exist (a torch >= 2.12 config). - test_per_tensor_quant_matches_native[dynamic] fails because AITER’s dynamic scale uses fp8 max 240 while native uses 224, landing on different fp8 grids. Fix: Schema tests now use the shared fp8-safe opcheck helper, which patch…

• 84c62e1 #46535 — [Model Runner V2][MM] Support EVS (#46535)
  • 作者: Nick Hill | +245/-50 | 10 个文件

  This also changes the Qwen VL recompute_mrope_positions methods to accept input_ids as a device tensor rather than python list, to avoid cpu roundtrip.

• 563c628 #46607 — [XPU] bump up vllm_xpu_kernels to v0.1.10.1 (#46607)
  • 作者: Kunshang Ji | +1/-1 | 1 个文件

  bump up vllm_xpu_kernels to v0.1.10.1 ## Test Result —

• 62890e2 #46467 — Fix duplicated logging when loading a corrupt or partial video (#46467)
  • 作者: Tae Jeong | +6/-6 | 1 个文件

  Loading a corrupt/partial video logged one WARNING per broken frame in OpenCVVideoBackend, duplicating the single summary line. Demote the per-frame messages to debug; keep the summary at warning. Verbosity-only. ## AI assistance Prepared with AI assistance (Claude); reviewed by submitter. — - [v] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”. - [v] …

• cf9fd64 #46284 — Fix KV offload request-finished lifecycle contract (#46284)
  • 作者: Rui Yin | +336/-114 | 5 个文件

  Fixes #46027. This aligns the KV offload on_request_finished() lifecycle contract across layers: - OffloadingConnectorScheduler now calls OffloadingManager.on_request_finished() when the request stops and no more GPU<->primary submit-side work will be issued, while retaining request state for later complete_store() / complete_load() callbacks from already-submitted jobs. - TieringOffloadingManager…

• d4448b5 #45973 — [XPU][Docker] switch to ubuntu 24.04 as base image (#45973)
  • 作者: Kunshang Ji | +21/-11 | 2 个文件

  switch base image to ubuntu 24.04, this can reduce image size ## Test Result —

• 160c80a #46582 — [Rust Frontend] Raise frontend JSON body limit (#46582)
  • 作者: Roy Wang | +44/-0 | 2 个文件

  The Rust frontend OpenAI routes use Axum’s JSON extractor, which has a 2 MiB default request body limit. Large valid requests from 1M context length support models can be rejected before validation with Failed to buffer the request body: length limit exceeded. This raises the router JSON body limit and adds a regression test for a request larger than Axum’s default. ## Duplicate Check Searched ope…

• f237e16 #45053 — [KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
  • 作者: Martin Hickey | +176/-465 | 13 个文件

  Makes KV transfer direction explicit and removes the medium routing dispatcher in the offloading worker path. Today, every transfer goes through a concrete OffloadingWorker in kv_offload/worker/worker.py that routes each job by (src_medium, dst_medium) to a registered handler. The OffloadingHandler.transfer_async is a single direction agnostic entry point. But each worker owns exactly one offloade…

• d20dbf9 #46412 — [Mooncake] Only check and store new KV cache range (#46412)
  • 作者: Wei Zhao | +487/-60 | 6 个文件

  This PR optimizes CPU overhead in Mooncake Store KV cache store path. Existing code checks KV cache range from the beginning in every store call. This PR only checks the range for the newly generated KV cache during chunked prefill. This saves various CPU overhead, including batch_is_exist lookup. ### Performance benchmark Deepseek v4 on 8xB300 ## Test Result —

• ede54b9 #46555 — set AttentionCGSupport.UNIFORM_BATCH for fa2 on xpu (#46555)
  • 作者: Xinyu Chen | +1/-1 | 1 个文件

  FA2 on XPU will decide grid size by max_query_len, it won’t support ALWAYS with mix-prefill-decode batch. ## Test Result —

• dc0d318 #46189 — [Attention] Add FLASH_ATTN_MLA_SPARSE backend for Hopper sparse MLA (#46189)
  • 作者: Dakai An | +292/-0 | 4 个文件

  This PR adds a Python-only FLASH_ATTN_MLA_SPARSE backend for sparse MLA models on Hopper GPUs. The backend reuses vLLM’s existing FlashAttention MLA wrapper and sparse MLA index conversion path, then calls flash_attn_varlen_func(…, fa_version=3) for BF16 KV-cache sparse decode. The tested GLM-5.1-FP8 checkpoint has: - num_attention_heads = 64 - kv_lora_rank = 512 - qk_rope_head_dim = 64 - compre…

• d7c1821 #45810 — [Model][MiniMax-M3] Add pipeline parallelism support (#45810)
  • 作者: soaringk | +124/-49 | 3 个文件

  Adds pipeline-parallelism (PP) support to MiniMax-M3. Fixes #45455. Relates to #45668. - Implement SupportsPP for MiniMaxM3SparseForCausalLM and MiniMaxM3SparseForConditionalGeneration. - Add PP first/last-rank handling for embed_tokens, final norm, and lm_head. - Thread intermediate_tensors through the MiniMax-M3 text and VL wrappers. - Update docs/models/supported_models.md for MiniMax-M3 PP sup…

• 4cd1a84 #46362 — [Model] Remove BaiChuanForCausalLM and BaichuanForCausalLM (#46362)
  • 作者: Tiezhen WANG | +2/-536 | 10 个文件

  • Remove BaiChuanForCausalLM and BaichuanForCausalLM model architectures (Baichuan 7B/13B/Baichuan2) - Adds both architectures to _PREVIOUSLY_SUPPORTED_MODELS ## Removed files - vllm/model_executor/models/baichuan.py — custom model implementation (~490 LOC) - examples/template_baichuan.jinja — chat template - rust/src/chat/tests/templates/vllm_examples/template_baichuan.jinja — Rust snapshot ## No…

• 191826e #46550 — [CI/Build] Fix topk histogram build on SM75 (#46550)
  • 作者: Mohammad Miadh Angkad | +13/-4 | 1 个文件

  Fix release build failure introduced by #43008, where redux.sync.add.u32 was emitted for sm_75 even though it requires sm_80+ CI/Build

• 489abad #44124 — feat: support to OpenMOSS-Team (#44124)
  • 作者: hurukawa | +2926/-2 | 11 个文件

  Closes #43431. Also manually smoke-tested four MOSS-Audio models with both audio_url and input_audio. E2E audio input: - Source: existing vLLM public test asset from vllm/assets/audio.py - Asset: AudioAsset(“winning_call”), resolved by vLLM from s3://vllm-public-assets - File: multimodal_asset/winning_call.ogg - URL: https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/winning_ca…

• 96de8bb #46549 — [MoE] Free unused MXFP4 scales in OAI Triton Backend (#46549)
  • 作者: Woosuk Kwon | +20/-4 | 2 个文件

  When using the OAI Triton backend, the original mxfp4 block scales are swizzled into the precision configs and are no longer read by the kernel, so they can be dropped. This eliminates the extra GPU memory (~6 GB) consumed by the Triton MoE backend.

• 556bc4e #46568 — Upgrade tpu-inference to v0.23.0 (#46568)
  • 作者: Sting Lin | +1/-1 | 1 个文件

  Upgrade tpu-inference to latest stable release v0.23.0 Verified on tpu-inference CI. ## Test Result Success. —

• 05a0cab #46188 — [Mooncake] Optimize lookup pool key string construction (#46188)
  • 作者: Wei Zhao | +57/-31 | 3 个文件

  This PR optimizes the Mooncake store lookup key construction by building TP/PP prefixes in advance and reuse them, avoiding KeyMetadata instantiation and reducing f-string construction overhead. edit: much of the optimization is already covered by https://github.com/vllm-project/vllm/pull/45969, this PR adds some minor improvment. ## Performance Benchmark DeepSeek v4 TP4 on 4 x GB300: ## Test Resu…

• 7ee4d22 #46533 — [Spec Decode] Reject placeholder (-1) draft tokens in rejection sampler (#46533)
  • 作者: Nick Hill | +75/-3 | 4 个文件

  The rejection sampler kernels gathered target/draft probabilities by the draft token id. For a placeholder draft id (-1, used for spec positions that must not be verified – structured-output grammar-invalid drafts and decode-side padded drafts) this read can be out of bounds in the non-greedy path and could in theory spuriously accept the placeholder, emitting -1 as a real token. Always reject a …

• ce9f640 #46360 — [Rust Frontend] Pass effective reasoning_parser_kwargs for structured output (#46360)
  • 作者: Bugen Zhao | +165/-34 | 19 个文件

  Follow-up to #46359. Pass the effective chat-template kwargs to engine-side reasoning parsers when structured output is enabled. Python forwards reasoning_parser_kwargs={“chat_template_kwargs”: …} so the structured output manager can initialize reasoning parsers with the same template controls used while rendering the prompt. This PR wires the Rust path the same way: - Renderers return the effec…

• 4ed8eaa #46057 — [Rust Frontend] Integrate xgrammar-structural-tag for strict and required tool calling (#46057)
  • 作者: Bugen Zhao | +466/-81 | 23 个文件

  Integrates xgrammar-structural-tag into the Rust frontend tool-calling path: - Adds the xgrammar-structural-tag dependency and exposes structural-tag model mappings from supported Rust tool parsers. - Applies structural-tag guided decoding in the chat output processor for tool_choice=required, named function choices, and tool_choice=auto with strict tools. - Lowers tool_choice=required and named f…

• 6af0559 #46532 — [Core][DP] Throttle prefills based on local prefill work (#46532)
  • 作者: Nick Hill | +42/-20 | 2 个文件

  Small follow-on to https://github.com/vllm-project/vllm/pull/44558. The DP prefill cadence deferred a waiting request iff request.num_computed_tokens == 0. That over-throttled requests whose whole prompt was already computed (full prefix-cache hit, or a fully remote-KV-loaded decode on the P/D decode side) – those are single-token decode steps with no local prefill to defer – while a request res…

• 0a3e2db #46428 — [Optimization] Skip DP padding tokens in MoE (#46428)
  • 作者: Woosuk Kwon | +154/-1 | 9 个文件

  For DP/EP with CUDA graphs, a batch can contain a large number of padding tokens. Today, these padding tokens are handled the same way as real tokens, introducing unnecessary compute overhead. For example, if one DP rank has a 512-token prefill while the other ranks are idle, every DP rank pads its input batch to 512 tokens—the maximum across all DP ranks, since 512 falls within the PW CUDA graph …

• abc3313 #46530 — [CI Test] Mark batch invariance test flaky (#46530)
  • 作者: Wentao Ye | +1/-0 | 1 个文件

  Thanks for the context from @mgoin , this batch invariance test seems to be flaky in certain node. Locally test passes as well pytest tests/v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle[FLEX_ATTENTION] ============================ 1 passed, 19 warnings in 46.96s ============================= Maybe worth a deep dive, but let’s fix CI fir…

• 0775b88 #45836 — [NVFP4 MoE/Deepseek V4] Marlin: wire SwiGLU clamp + allow it for clamped models on non-Blackwell (#45836)
  • 作者: Mike G | +4/-0 | 2 个文件

  Address https://github.com/vllm-project/vllm/issues/45859. NVFP4 MoE models that declare a SwiGLU clamp (config.swiglu_limit, e.g. DeepSeek-V4) cannot be served on any non-Blackwell GPU (SM80/SM89/SM90). select_nvfp4_moe_backend() raises: When swiglu_limit is set, the selector restricts candidates to NVFP4_BACKENDS_WITH_CLAMP, which previously contained only FLASHINFER_TRTLLM (Blackwell/SM100-only…

• 7c2e084 #46517 — [Docker] Remove redundant flashinfer download-cubin step (#46517)
  • 作者: Michael Goin | +0/-7 | 1 个文件

  The flashinfer-cubin wheel — a default CUDA dep in requirements/cuda.txt since #37233 — already ships all precompiled cubins into the package directory that FlashInfer’s runtime loader reads from (jit/env.py:_get_cubin_dir resolves to the package dir when flashinfer-cubin is installed). The flashinfer download-cubin step in vllm-base re-downloads the same files from NVIDIA’s artifactory **uncondit…

• ef361de #46435 — [Model Runer V2][DFlash] Fix lm head sharing for dflash (#46435)
  • 作者: Giancarlo Delfin | +4/-7 | 1 个文件

  Context When serving with Model Runner V2, mimo + dflash has nearly 0% acceptance rate. This was caused by the draft model’s LM head not being set, resulting in garbage/zero hidden states. The reason the LM head wasn’t being set was because DFlashQwen3ForCausalLM was initializing the draft_id_to_target_id property, despite the DFlash model not owning an LM head. This caused this check to be Fals…

• acce57d #44514 — Deprecate old FP8 online MoE quantization class (#44514)
  • 作者: Yan Ma | +19/-137 | 3 个文件

  Deprecate old FP8 online quantization classes Fp8OnlineLinearMethod and Fp8OnlineMoEMethod, as it can be replaced by new online quantization frontend. pytest -s -v tests/quantization/test_fp8.py ## Test Result —

• 6691f08 #45892 — [Minimax-M3] BF16/FP8 Indexer using MSA (#45892)
  • 作者: Yongye Zhu | +1048/-104 | 13 个文件

  Integrating MSA indexer prefill kernel. Decode remain triton for performance reason Minimax M3 gsm8k, AIME25, GPQA in TP4 ## Test Result with FP8 attention and FP8 Indexer cache. gsm8k: 92 GPQA-D: 93.6 AIME25: 92.5 —

• 40e5522 #46197 — [Docs] Add Qwen3 forced alignment online example (#46197)
  • 作者: Taneem Ibrahim | +213/-0 | 1 个文件

  This PR adds an online /pooling example for Qwen/Qwen3-ForcedAligner-0.6B. Relevant issue: #38805 The motivation is that users can already run Qwen3 ForcedAligner through vLLM’s offline pooling API, but the online /pooling request shape is less obvious because it needs both audio input and reference text. This example gives users a simple ready to use script that shows the full request format, inc…

• f3410b3 #45404 — fix(moe_wna16): access tp_size via moe_config for RoutedExperts compatibility (#45404)
  • 作者: Willow Lopez | +4/-3 | 1 个文件

  Fix AttributeError: ‘RoutedExperts’ object has no attribute ’tp_size’ when loading AWQ MoE models (Qwen3-Coder-30B-A3B, Qwen3-30B-A3B) with TP≥2. When AWQMoeMarlin falls back to Moe WNA16 kernels for MoE layers, moe_wna16_weight_loader accessed layer.tp_size directly. However, RoutedExperts stores this value at layer.moe_config.tp_size — self.tp_size is never set in RoutedExperts.init. ## Fix …

• 275b431 #39896 — [MyPy] Fix mypy for vllm/benchmarks (#39896)
  • 作者: Martin Hickey | +296/-150 | 24 个文件

  Part of #26533 pre-commit run -a –hook-stage manual mypy-3.10 Before: After: Run mypy for Python 3.10………………………………………….Passed ## Test Result Run with error and produce output similar to the following:

• 547d2c4 #44763 — Add weights padding for fp8 per-block online quantization (#44763)
  • 作者: Yan Ma | +55/-0 | 1 个文件

  FP8 per-block online quantization use group size 128 thus it requires MoE hidden/intermediate dimensions per rank should be divisible by this value. Take triton moe backend for example, when run Qwen/Qwen3-30B-A3B using tp=4, the dimensions is 768/4=192 and will cause below error: ## Test Result gsm8k lm_eval + tp=2(w/o padding): gsm8k lm_eval + tp=4(w/ padding): —

• 9f6f296 #46494 — [CI/Build] Remove BaiChuanForCausalLM from the LoRA test (#46494)
  • 作者: Jee Jee Li | +12/-9 | 1 个文件

  Address https://github.com/vllm-project/vllm/pull/46362#discussion_r3452479264 ## Test Result —

• e51e700 #45715 — [LoRA] Gate all_gather on fully_sharded_loras inside _mcp_apply; rewrite regression test (#45715)
  • 作者: lcheng | +5/-5 | 1 个文件

  Problem _mcp_apply() unconditionally calls tensor_model_parallel_all_gather() on the shrink buffer. With fully_sharded_loras=False, lora_A is replicated at full max_lora_rank per rank, so all_gather inflates the rank dim to max_lora_rank * tp_size before add_expand, which expects max_lora_rank. This produces incorrect LoRA output silently (no crash) under TP > 1. Conditions: enable_lora=True, t…

• 9f51178 #46135 — [HARDWARE][POWER] Enable fp16 support for PowerPC (#46135)
  • 作者: Rukhaiya2004 | +333/-69 | 6 个文件

  Enable FP16 (half precision) inference support for PowerPC systems.

• 1bf149f #46457 — Filter Pydantic-internal markers from validation error param (#46457)
  • 作者: Muhammad Fawaz | +115/-2 | 2 个文件

  Follow-up to #46038 (param fallback for plain validation errors), scoped by the same fuzz run that surfaced the file-path leak fixed in #46415 (sanitize-validation-error-paths). #46038 added a fallback in validation_exception_handler that joins the Pydantic error loc tuple with . to populate param for plain (non-VLLMValidationError) validation errors. For union-typed or internally-wrapped fields, …

• 091bc10 #45959 — [KV Offloading] Add tiering metric plumbing (#45959)
  • 作者: Srinivas Krovvidi | +184/-13 | 5 个文件

  • add default build_metric_definitions() and get_stats() hooks to SecondaryTierManager - plumb secondary tier metric definitions through TieringOffloadingSpec - aggregate primary and secondary tier stats through TieringOffloadingManager - add a test-only secondary tier to validate labeled metric definition and stats plumbing

• d32575a #46332 — [ROCm][P/D] Support MoRIIO heterogeneous TP fan-in (#46332)
  • 作者: Tan Pin Siang | +544/-33 | 6 个文件

  This PR builds on the MoRIIO typed control-message support merged in #46290. This PR adds MoRIIO READ-mode support for heterogeneous prefill/decode TP sizes, e.g. P4/D8 and P8/D4. - Add remote TP rank mapping for local_tp != remote_tp. - Use the mapped TP rank for MoRIIO handshake and READ release notify ports. - Send READ release ACKs as typed msgpack release messages with consumer_tp_size. - Cou…

• 83fa302 #46463 — fix(security): prevent infinite loop in split_audio with NaN audio sa… (#46463)
  • 作者: Juan Pérez de Algaba | +43/-2 | 2 个文件

  find_split_point returns 0 when all audio samples are NaN because NaN comparisons always evaluate to False under IEEE 754. T his causes split_audio to never advance its loop cursor, permanently hanging the worker thread. Fix by defaulting quietest_idx to start_idx instead of 0, skipping NaN energy values, and adding a progress guard in split_audio that falls back to the hard chunk boundary when th…

• 901a3b0 #46441 — fix gpt_oss pp>1 with ep (#46441)
  • 作者: Qiming Zhang | +1/-1 | 1 个文件

  Bug: In GptOssModel.load_weights(), get_ep_group().rank returns the global rank across all workers, but the code uses it to slice expert weights as if it were the rank within the EP group. With pipeline parallelism, PP stage 1 workers have global ranks offset by pp_rank * tp_size (e.g., ranks 2 and 3 for a 2-stage pipeline). This causes ep_rank_start = 2 * experts_per_rank, which is out-of-bou…

• 2d721ab #46348 — [Rust Frontend] Align Rust allowed_token_ids validation with Python (#46348)
  • 作者: Reid | +138/-27 | 6 个文件

  The Rust OpenAI frontend currently accepts allowed_token_ids: [] through the HTTP completions and chat completions paths. The request is lowered as Some(vec![]), and the existing token-id validation only checks for out-of-vocab IDs, so the empty list is accepted. Python SamplingParams rejects this case with: This PR aligns the Rust frontend with the Python behavior by rejecting allowed_token_ids: …

• accaa43 #46219 — [Rust Frontend] Support echo for token-ID completion prompts (#46219)
  • 作者: Reid | +275/-23 | 4 个文件

  Part of https://github.com/vllm-project/vllm/issues/44280 The Rust OpenAI completions endpoint rejected requests using both echo=true and token-ID prompts. Python accepts this flow and returns the decoded prompt text as the echoed prefix. This change aligns the Rust frontend with Python behavior: - echo=true with token-ID prompts is accepted. - The engine still receives the original token IDs. - N…

• a04654d #46452 — Doc: fix missing GLM-5.x in supported models (#46452)
  • 作者: Sunny Yuan | +1/-0 | 1 个文件

  add GLM-5.x to supported models table FIX https://github.com/vllm-project/vllm/issues/46442

• 25bc3be #46359 — [Rust Frontend] Correct --reasoning-parser semantics (#46359)
  • 作者: Bugen Zhao | +232/-43 | 7 个文件

  Align Rust frontend parser-selection semantics with the path that supplies the arguments. Python-supervised bootstrap receives runtime args from Python through JSON. In that path, omitted tool_call_parser and reasoning_parser now deserialize to none, matching Python CLI semantics where parser support stays disabled unless Python explicitly validated and forwarded a concrete parser name. Rust manag…

⚡ Performance

• 7f99e80 #46425 — [Perf][ThinkingBudget] reduce search space for thinking tokens (#46425)
  • 作者: Walter Beller-Morales | +61/-4 | 2 个文件

  Currently the thinking token budget logic uses _find_last_sequence_index to scan the entire generated sequence on every single generated token. This causes the scan time to blow up on longer generations because it’s basically a O(n^2) nested loop. One way to fix this time complexity is to keep track of the sequences that have already been scanned in the TB state and only search for new tokens. Thi…

• 061043e #46353 — [CPU][Perf] Accelerate unquantized MoE for AArch64 (#46353)
  • 作者: Fadi Arafeh | +672/-32 | 15 个文件

  Accelerate unquantized MoE for AArch64 - Enable FusedMoE kernel for AArch64 - Implement AdvSIMD BFMMLA interface to accelerate w13 and w2 GEMMs - Extend generic micro kernel interface and MoE kernel to support packing input matrix - Abstract sleef.h includes and tanh symbol for x86 under the AVX vectorizer class ## Performance 1.96x higher throughput for gpt-oss and 2.18x higher throug…

• 0bc479e #46542 — [Perf][LoRA] Replace O(n) list.index() with a dict in convert_mapping (#46542)
  • 作者: Lynn | +11/-3 | 1 个文件

  convert_mapping (in vllm/lora/punica_wrapper/utils.py) builds the per-token LoRA index lists by calling lora_index_to_id.index(…) for every prompt token and every batch token: Each list.index() is an O(num_loras) linear scan, so the construction is O(num_tokens × num_loras) every time LoRA metadata is rebuilt — which is on the per-forward path via update_metadata. The code already flagge…

• 52fbe12 #46543 — [Perf][Multimodal] Avoid building a full timestamps list in video frame sampling (#46543)
  • 作者: Lynn | +2/-4 | 1 个文件

  The GLM-4V / GLM video frame-sampling backends (GLM46VVideoBackend and GLMGAVideoBackend in vllm/multimodal/video.py) build a Python list with one float timestamp per source frame: and then only ever read it as timestamps[frame_index] inside the sampling loop. Two issues: 1. The list is O(total_frames_num) to allocate, but is never read when the total_frames_num < extract_t branch is taken…

• 855cd4d #43008 — [Perf][DSv4/DSv3.2] Add cluster-cooperative topK kernel for low-latency scenarios (#43008)
  • 作者: Roberto L. Castro | +1568/-45 | 10 个文件

  Adds a cluster-cooperative topK kernel for low-latency cases, which uses TMA and DSMEM, meaning that SM90+ arch is required. This new kernel has been extensively tuned to cover the whole low-latency regime (i.e. bs ≤32) with a cluster-level cooperation via cluster.sync() and distributed SMEM histogram reduction, eliminating the complexity of the persistent scheduler and multi-CTA spin-barrier coor…

• 3cc871a #46422 — [Perf] Skip detokenization in online beam search (#46422)
  • 作者: Guy Stone | +1/-0 | 1 个文件

  What & why Online beam search builds an internal per-step SamplingParams requesting logprobs = 2 * beam_width. The engine then detokenizes every one of those 2 * beam_width logprob token ids into strings on every decode step (LogprobsProcessor → tokenizer.decode). Beam search never uses those strings — it ranks candidates by cum_logprob and detokenizes only the final selected sequences itse…

• 20b5af5 #43673 — [ROCm][Perf] DSv3.2: fuse MLA Q concat+fp8-quant in forward_mqa (#43673)
  • 作者: frida-andersson | +12/-7 | 1 个文件

  On the ROCm sparse-MLA fp8-KV decode path, the query tensor was being built by two back-to-back HBM-bound kernels: ConcatMLAQKernel<bf16, 512> (concatenate q_nope + q_pe into bf16, ~5.2 µs) followed by scaled_fp8_quant (~5 µs). Both are pure copy-with-scale, no compute. This PR routes forward_mqa through layer._decode_concat_quant_fp8_op(ql_nope, q_pe, q_scale) (already used by MLACommonImpl), whi…

🧪 CI/Tests

• 2801b11 #45914 — [Test] Pin block_size in auto-fit max_model_len test (#45914)
  • 作者: Liangliang Ma | +4/-2 | 1 个文件

  The kv_cache_memory_bytes=1MB budget assumed CUDA’s default block_size=16. On platforms whose default block_size is larger (e.g. XPU defaults to 64) a single KV block already exceeds 1MB, so auto-fit fails with ’not enough GPU memory available to serve even a single token’. Pin block_size=16 in the test so the budget is independent of the platform default.

• 549c707 #46580 — [ROCm][CI] Skip the MoE Marlin tile-padding helper assertion (#46580)
  • 作者: Andreas Karatzas | +4/-0 | 1 个文件

  Skip the MoE Marlin tile-padding helper assertion on ROCm, where production code intentionally keeps MoE Marlin disabled and falls back to ROCm/native MoE paths. test_check_moe_marlin_supports_layer_padding was added in #45703 with CUDA Marlin selection semantics, but the helper it exercises still returns False immediately on ROCm. The AMD kernel-quantization lane was already printing this failure…

• bcbeaac #46537 — [ROCm][CI] Stage C-II of gating additional test groups (#46537)
  • 作者: Andreas Karatzas | +96/-15 | 10 个文件

  Mirrored/Gated Test Groups mi300_1: - Benchmarks CLI Test - Pytorch Nightly Dependency Override Check - EPLB Algorithm - Multi-Modal Accuracy Eval (Small Models) - Model Executor - Language Models Tests (Standard) - Language Models Tests (Extra Standard) %N mi300_2: - Distributed DP Tests (2 GPUs) - Metrics, Tracing (2 GPUs) - V1 e2e (2 GPUs) Agent Count 8 x mi300_1, 3 x mi300_2

• 84f1337 #46164 — [CI] Fix test_auto_gptq on ROCm CI (#46164)
  • 作者: fxmarty-amd | +2/-2 | 1 个文件

  Update test_auto_gptq to reflect the supported ROCm behavior for explicit Marlin requests on GPTQ checkpoints. quantization=“marlin” for GPTQ checkpoints is accepted by AutoGPTQConfig.override_quantization_method() and normalized to “auto_gptq”. On ROCm, this falls back to the generic AutoGPTQ path, which is supported. The test previously expected this case to fail on non-CUDA platforms, but ROCm …

• b28103e #46520 — [ROCm][CI] Shard LM Eval Qwen3-5 Models (B200-MI355) in AMD CI (#46520)
  • 作者: Micah Williamson | +5/-3 | 2 个文件

  LM Eval Qwen3-5 Models (B200-MI355) is a long running test group that is timing out in AMD CI after 2 hours. We need to extend the timeout, and I added sharding since it is such a long TG Example failure: https://buildkite.com/vllm/amd-ci/builds/9851/list?jid=019ef3b5-76ba-4b1a-b69a-5a7cb2468ef3&tab=output

• e368415 #46142 — [AMD][OCP MX][CI] Fix tests to not dispatch on UNFUSED_TRITON backend on MI300, improve w_mxfp4_a_fp8 emulation support (#46142)
  • 作者: fxmarty-amd | +74/-17 | 7 个文件

  This PR: 1. Forces OCP MX tests to use MOE emulation code path on MI250, MI300, MI325. This should have been done following https://github.com/vllm-project/vllm/pull/41436/changes#r3237495909 but was not done in #41436 2. Improves W-MXFP4-A-FP8 emulation, so that the test test_gpt_oss_attention_quantization and amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8 is meaningfully running on MI300. 3. Rem…

• ceae5bc #45219 — [ROCm][CI] Fix nixl tests (#45219)
  • 作者: Andreas Karatzas | +210/-15 | 8 个文件

  Fixes NIXL PD accuracy for deepseek-ai/deepseek-vl2-tiny on ROCm by deriving a valid head size when the loaded HF text config exposes head_dim=0. deepseek-vl2-tiny is not using the DeepSeek MLA checkpoint layout: layer 0 has separate q_proj, k_proj, and v_proj weights, and no kv_a_proj_with_mqa / kv_b_proj weights. Its raw config also has kv_lora_rank=None and zero qk/v MLA dims. However, after co…

• fd50a66 #46160 — [CI][ROCm] Skip unsupported test cases on ROCm (#46160)
  • 作者: fxmarty-amd | +34/-2 | 2 个文件

  Skip unsupported test cases on rocm. Skipping them for now. - CUDA_VISIBLE_DEVICES=0 pytest tests/quantization/test_compressed_tensors.py -s -vvvvv -k “test_compressed_tensors_nvfp4 and args0” - pytest tests/quantization/test_modelopt.py -s -vvvvv -k “test_modelopt_mixed_precision_dispatches_w4a16_layer” ## Test Result test_compressed_tensors_nvfp4 and args0 validating the correct error message is…

• 84586c9 #46410 — [ROCm][CI] fix fp8 range in vit_fp8_quant (#46410)
  • 作者: Divakar Verma | +4/-2 | 1 个文件

  Fix fp8 min/max range for FNUZ platform Fixes: kernels/core/test_vit_fp8_quant.py::test_quantize_contiguous

• 568874f #45869 — [ROCm][CI] pass merge-base to container for python-only wheel metadata (#45869)
  • 作者: Divakar Verma | +76/-4 | 3 个文件

  Pass merge-base with origin/main into the ROCm test container as VLLM_STANDALONE_MERGE_BASE, so python_only_compile.sh can resolve wheels.vllm.ai metadata when /vllm-workspace is an artifact layout without .git. Keep the in-script git fallbacks for checkouts that still have a usable repo.

• 2aaaf3f #46260 — [ROCm][Test] Fix stale test_gfx950_moe MXFP4 oracle tests (#46260)
  • 作者: Spandan Tiwari | +23/-8 | 1 个文件

  The gfx950 (MI355) MXFP4 MoE oracle tests in tests/quantization/test_gfx950_moe.py had drifted from the product API and no longer ran. This restores them. Three independent breakages, all from product evolution that the test wasn’t updated for: 1. FusedMoEConfig signature — intermediate_size is now a required field and intermediate_size_per_partition is derived in post_init. The test helpe…

• 156b126 #46431 — [ROCm][CI] Skip Quark mxfp4 tests unless Quark version is compatible with Torch version (#46431)
  • 作者: Micah Williamson | +34/-8 | 2 个文件

  We’re seeing an issue in Quark with Torch 2.11. https://buildkite.com/vllm/amd-ci/builds/9797/list?jid=019ef091-cd38-413c-b6af-ca6920a07a81&tab=output The fix will be in Quark 0.12.0 https://github.com/amd/Quark/issues/34. For now we will skip these tests until Quark 0.12.0 is released.

🐛 Bug Fix

• 24d5186 #46339 — [Bugfix] Re-enable FP8 MoE on NVIDIA Thor (#46339)
  • 作者: Cyrus Leung | +3/-3 | 2 个文件

  Partially revert a change in https://github.com/vllm-project/vllm/pull/45277 which broke Qwen/Qwen3.5-35B-A3B-FP8 inference on NVIDIA Thor (SM101 for CUDA 12 and SM110 for CUDA 13). This parallels how cutlass_3x_gemm_sm100_fp8 is also enabled for this architecture. ## Test Result —

• 93ec645 #44483 — [Bugfix] Fix illegal memory access from a forward during a partial wake_up (#44483)
  • 作者: meihanc | +12/-4 | 1 个文件

  Fixes #44395. Resume scheduling only once the executor is fully resident. model_executor.is_sleeping stays True until all sleep tags are awake (Executor.wake_up clears it only when sleeping_tags is empty), so a weights-only wake keeps the scheduler paused; it resumes at the final/full wake. No behavior change on the full-wake path. ### Reproduction ### Test plan & results Validated on **4× A100-80…

• f1a6703 #46220 — [Bugfix][Config] Keep pydantic validation for fields with a TYPE_CHECKING Literal alias (#46220)
  • 作者: Ting SUN | +6/-4 | 3 个文件

  Follow-up to #45196, addressing @hmellor’s review on load_format. A field annotated str | SomeAlias, where SomeAlias is a Literal imported only under TYPE_CHECKING and falling back to = Any at runtime, collapses to Any at runtime, so pydantic skips validation. #45196 worked around this by retyping load_format to plain str (gaining validation but dropping the LoadFormats Literal); @hmellor asked to…

• 4c5bc41 #45956 — [Bugfix][Spec Decode] Fix probabilistic sampling for parallel drafting (#45956)
  • 作者: Benjamin Chislett | +17/-0 | 2 个文件

  In MRV1, Parallel drafting samples with a different shape than sampling_metadata expects. ## Testing This enabled me to run experiments using probabilistic acceptance, with decent results. Seems to be sampling appropriately now. No longer crashes.

• ac1fa74 #46495 — [Bugfix] Fix NemotronLayerNorm1P hardcoded cuda device type (#46495)
  • 作者: Michał Ganczarenko | +10/-8 | 1 个文件

  Fix NemotronLayerNorm1P using hardcoded device_type=“cuda” in autocast calls. _cast_if_autocast_enabled and NemotronLayerNorm1P.forward hardcoded device_type=“cuda”, causing self.weight + 1 to be placed on the wrong device index during AOT compilation with TP>1 on non-CUDA backends. This produces a cross-device mm crash at profile_run: Fix: derive device_type from x.device.type and thread it throu…

• e2bdc24 #45998 — [ROCm][Bugfix] Fix use_v2_model_runner inside Ray driver thread (#45998)
  • 作者: Micah Williamson | +23/-4 | 2 个文件

  This fixes an edge case demonstrated in AMD CI: https://buildkite.com/vllm/amd-ci/builds/9518/list?sid=019ec55c-2635-47ef-ad70-0a76f7edab96&tab=output Further up in the logs, we see this conflicting information: So we’ve got a process where use_v2_model_runner evaluated to True, and one worker process where it evaluated to False. What’s happening here is that in the main Ray thread, HAS_TRITON eva…

• e48f2aa #46525 — [Bugfix][Frontend] Emit a content block for empty Anthropic completions (#46525)
  • 作者: Maxwill Lin | +45/-0 | 2 个文件

  An empty completion (immediate EOS / max_tokens=1 / content filter) is serialized by messages_full_converter as content: []. The Anthropic Messages API requires at least one content block, and strict SDK clients reject an empty content array. Emit one empty text block in that case. Split out of #46321 (the message-ordering fix) so the two independent fixes can be reviewed separately. tests/entrypo…

• 80e5117 #44434 — [ROCm][Bugfix][Perf] enable shared expert fusion for Qwen3.5 (#44434)
  • 作者: Nico Holmberg | +48/-4 | 2 个文件

  The existing FSE (Fused Shared Expert) support (#39280) works for Qwen3-Next but fails on Qwen3.5 models because qwen3_5.py’s load_weights does not remap shared expert checkpoint weights to the fused expert slot. This causes shared expert weights to silently fail to load, producing garbage output when VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. ## Root Cause When FSE is enabled, Qwen3NextSparseMo…

• 6617db1 #46308 — [Bugfix][Frontend] Emit non-ASCII tool-call arguments without \uXXXX escapes (#46308)
  • 作者: Maxwill Lin | +75/-9 | 6 个文件

  The xlam, hunyuan_a13b, and seed_oss tool parsers serialize tool-call arguments with json.dumps(…) at its default ensure_ascii=True, so non-ASCII argument values (Chinese, emoji, …) are emitted as \uXXXX escapes instead of literal UTF-8. This adds ensure_ascii=False at the argument-serialization sites, matching the convention used across the other tool parsers — and seed_oss’s own non-streamin…

• 899d72a #45389 — [Bugfix][ToolParser] Handle braces in required tool streaming strings (#45389)
  • 作者: Ting SUN | +78/-44 | 2 个文件

  Related: #41111. In required tool_choice streaming, vLLM trims streamed tool-call JSON by counting the wrapper’s bracket level. The counter currently treats { and } inside JSON string values as structural braces, so a valid generated argument such as {“city”: “a } b”} can be emitted to the client as malformed function.arguments. This PR makes that bracket scan JSON-string aware. It also keeps the …

• 0d4d164 #46492 — [Bugfix] Allow flashinfer_cutlass as a clamped NVFP4 MoE backend (#46492)
  • 作者: Gabriel Wu | +1/-0 | 1 个文件

  Models that set a SwiGLU clamp (swiglu_limit) – e.g. DeepSeek-V4-Flash NVFP4 – restrict NVFP4 MoE backend selection to NVFP4_BACKENDS_WITH_CLAMP, which previously contained only FLASHINFER_TRTLLM. The TRTLLM NVFP4 fused MoE only supports Blackwell datacenter GPUs (capability family 100 / SM100), so on SM120 (e.g. RTX PRO 6000) selection fails with “No NvFp4 MoE backend supports the deployment co…

• 68afd78 #46203 — [Bugfix][ROCm] Fix cumem sleep and teardown (#46203)
  • 作者: peizhang56 | +130/-18 | 7 个文件

  Free ROCm sleep-mode VRAM by releasing and re-reserving the virtual address placeholder, and track asleep allocations so final frees avoid double unmap/release. Release kept-alive cumem MemPool objects from allocator-owned shutdown while pluggable allocator wrappers remain alive, avoiding interpreter-finalization crashes without per-test teardown hooks. Fix two ROCm cumem sleep-mode issues found i…

• d8e422c #45718 — [Bugfix] Parse MiniMax M3 streaming reasoning by text markers (#45718)
  • 作者: Rui Yin | +357/-67 | 2 个文件

  Fixes #45687. This reopens the MiniMax M3 streaming reasoning fix against main. The earlier PR #45713 targeted m3_release, but that base branch was deleted after MiniMax M3 landed in main via #45381, so GitHub closed the PR automatically. MiniMax M3 streaming reasoning markers may be generated as visible marker text even when the tokenizer exposes mm:think / </mm:think> as vocab entries. The pre…

• f4d5f73 #45818 — [Bugfix]: Fix unquantized gpt-oss weight loading broken by FusedMoE r… (#45818)
  • 作者: Priyansh Jain | +5/-1 | 1 个文件

  The FusedMoE/MoERunner inversion refactor (#41184) moved expert weight parameters into a nested RoutedExperts submodule, renaming params from …mlp.experts. to …mlp.experts.routed_experts.. The unquantized loading path (_load_weights_other) still indexed params_dict with the raw checkpoint names, causing a KeyError when loading standard (non-mxfp4/non-quark) gpt-oss checkpoints. Route the…

• f59db63 #45048 — [Bugfix] GPT-OSS Autodrop reasoning in Response API and cleanup (#45048)
  • 作者: yzong-rh | +137/-46 | 5 个文件

  Coauthor: will-deines Extracted fix for Bug 2 from https://github.com/vllm-project/vllm/pull/35907 Harmony prompts should drop reasoning from completed turns, but keep reasoning from the current in-progress turn so tool-calling can continue correctly. Before this change, Chat Completions had two analysis-drop paths: vLLM’s own auto_drop_analysis_messages(), which was correct, and the Harmony encod…

• 2a675a7 #44361 — [Bugfix] Responses API assistant EasyInputMessageParam input (#44361)
  • 作者: yzong-rh | +93/-13 | 4 个文件

  Fix Responses API running into Pydantic errors on assistant EasyInputMessageParams with string content (anything like {“role”: “assistant”, “type”: “message”, “content”: “some string”}) The error was caused by assistant string content being incorrectly coerced into ResponseOutputMessage, causing Pydantic to try parsing each character of the string as an output content item. This PR fixes the issue…

• 7d47cff #46379 — [Bugfix][KV Offload] Fix swap_blocks_batch on the default stream (#46379)
  • 作者: Itay Etelis | +48/-1 | 2 个文件

  ops.swap_blocks_batch crashes with cuMemcpyBatchAsync failed at index 0 with error 1 (CUDA_ERROR_INVALID_VALUE) on the legacy default stream — cuMemcpyBatchAsync rejects it (handle 0 / CU_STREAM_LEGACY); per-thread-default and real streams are fine. Reproduced on H100 / CUDA 13.0 (driver 580.105.08): the legacy default stream triggers the error and a non-default stream succeeds through the same op…

• 3554ada #46069 — [CPU][Bugfix][Speculative Decoding] Accept USE_FP64_GUMBEL in CPU recovered-tokens sampler (#46069)
  • 作者: hillel.darshan | +3/-1 | 1 个文件

  #43150 (“Fix FP64 Gumbel precision coverage”, in v0.23.0) added USE_FP64_GUMBEL to sample_recovered_tokens_kernel and its caller in vllm/v1/sample/rejection_sampler.py — extending the GPU-only fp64-Gumbel work from #37798 to the shared sampler — but the CPU fallback shim _sample_recovered_tokens_kernel_impl in vllm/utils/cpu_triton_utils.py was not updated to accept it. So every CPU speculative-de…

• a46f3eb #46245 — [Bugfix][Model Runner V2] Preserve all allowed_token_ids in the logit bias kernel (#46245)
  • 作者: Ting SUN | +12/-0 | 2 个文件

  Under the V2 model runner, SamplingParams.allowed_token_ids can silently drop some of the allowed tokens. Generation then produces a token outside the allowlist, and when every allowed token happens to be dropped the whole logits row becomes -inf and the engine samples token id 0. Root cause is in the Triton _bias_kernel in vllm/v1/worker/gpu/sample/logit_bias.py. It applies the allowlist in three…

📖 Documentation

• 7dc0360 #44720 — [Doc] Document Qwen3.6 (dense + MoE) ViT CUDA graph support (#44720)
  • 作者: Nemani Harsha Vardhan | +2/-1 | 1 个文件

  Documents that Qwen3.6 (dense Qwen/Qwen3.6-27B and MoE Qwen/Qwen3.6-35B-A3B) is covered by the ViT-encoder CUDA-graph path. Per review feedback (@Isotr0py, @shen-shanshan): Qwen3.6 reuses the existing Qwen3.5 architecture — its config.json reports model_type: qwen3_5, so it is served by Qwen3_5ForConditionalGeneration / Qwen3_5MoeForConditionalGeneration, which already implement SupportsEncode…

🔩 Misc

• a2cb08b #46473 — [Misc][PD] Disable bidirectional xfer mode for NixlPushConnector (#46473)
  • 作者: Nicolò Lucchesi | +4/-0 | 1 个文件

  Bidirectional kv transfer with the newly added Push model is not supported, but there’s no validation right now

✨ New Feature

• 70749fd #40835 — [Feature] Triton INT4 per-token-head KV cache quantization (#40835)
  • 作者: JartX | +1463/-112 | 10 个文件

  Based on PR: Changed to only INT4_PER_TOKEN_HEAD INT4_PER_TOKEN_HEAD |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |—–|——:|—————-|—–:|———–|—|—-:|—|—–:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ | 0.912|± |0.0273| | | |strict-match | 5|exact_match|↑ | 0.91|± |0.0273|

• d86c66c #46167 — [Feat] Add runtime monitor for post-warmup CuTeDSL compilation (#46167)
  • 作者: Roberto L. Castro | +366/-172 | 7 个文件

  Warn mode (EngineCore pid=2386340) … [jit_monitor.py:71] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (EngineCore pid=2275828) WARNING 06-19 16:22:25 [jit_monitor.py:124] CuTeDSL JIT compilation during inference: FlashAttentionForwardSm100. This causes a latency spike; consider extending warmup to cover this shape/config. ### Error mode (Engin…

🖥️ Kernel

• 9d6fdc2 #46385 — [Kernel] GLM5 Router GEMM (#46385)
  • 作者: Jee Jee Li | +158/-32 | 4 个文件

  Borrrow idea from https://github.com/NVIDIA/TensorRT-LLM/pull/13740 ## Test Result ### GSM8K - main branch - this PR —

• 11b56b2 #46393 — [Kernel] Add FlashInferCutedslMxfp8LinearKernel (cute-dsl mm_mxfp8) (#46393)
  • 作者: Yongye Zhu | +87/-1 | 3 个文件

  Add an MXFP8 W8A8 linear GEMM that drives FlashInfer’s mm_mxfp8(…, backend=“cute-dsl”), sibling to the existing CUTLASS kernel. The cute-dsl backend consumes the same 1D swizzled F8_128x4 scales the CUTLASS path already produces, so weight/activation prep is identical and output is bit-identical; only the backend string and support gate differ. Gate to sm_100/sm_103 (matching FlashInfer’s suppor…

• 37a682d #45703 — [Kernel] Extend Marlin thread-tile padding to MoE (WNA16 + FP8/MXFP8) (#45703)
  • 作者: Michael Goin | +573/-29 | 7 个文件

  Follow-up to #45295, which added thread-tile padding to the dense Marlin paths and NVFP4 MoE. This extends the same mechanism to the remaining Marlin MoE preps: generic WNA16 (GPTQ/AWQ/compressed-tensors) and FP8/MXFP8. TP sharding can leave the rank-local intermediate size off a Marlin thread tile. Today WNA16 MoE rejects Marlin and falls back to slower kernels, and FP8/MXFP8 MoE crash at repack….

🦀 Rust Frontend

• 31ca950 #44285 — [Frontend] Split ServingRender into renderer and entrypoint. (#44285)
  • 作者: wang.yuqi | +1095/-879 | 26 个文件

  Following #41907 1. Remove the “OpenAI” prefix. > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. Remove the “OpenAI” prefix if this is not the official OpenAI API. - OpenAIServingRender -> ServingRender 2. ServingRender extends BaseServing and handles _check_model and more. 3. Split ServingRender into renderer an…