Compare commits

..
1640 Commits
Author SHA1 Message Date
minke.yu b419518774 deploy(benchb): bump new-stack image to g9i-est-td-148936bcb (hicache prefetch call fix) 2026-10-19 02:13:46 +08:00
minke.yu 148936bcb9 fix(hicache): drop stale completed_tokens arg in _check_hybrid_prefetch_result call
Merge artifact: upstream refactored the method to read
operation.completed_tokens internally (8 params), our call site kept
passing completed_tokens (9 args) -> TypeError at runtime on hybrid
prefetch completion, wedging the prefill engine under high L3 hit load.
2026-10-19 01:52:38 +08:00
minke.yu 6ceccac658 deploy: 阶段A/B 新旧基线 compose(4x2卡 standalone 并行 + 双 1P1D PD 并行)
- bencha: old/new × cp2/dp2 standalone,端口 30500-30800,新镜像独立 autotune cache
- benchb: 旧栈(31000/31200/31400+master 50051) 新栈(31100/31300/31500+master 50061),
  NVLink-intra 四件套,新版 dp2 加 --skip-server-warmup(上游 dspark+dp-attention
  warmup 死锁规避,真实流量路径不受影响)
2026-10-19 00:29:07 +08:00
minke.yu 1d2610bbc2 fix(mm): _collect_image_span_requests produce 3-tuple overlapping (idx,item,span_len)
Merged tree convention (PerImageRequestInfo.overlapping annotation and
upstream _assemble_per_image_chunk) is (idx, item, span_len); our span
collector still emitted 4-tuples (idx,item,start,end) from the pre-merge
version, crashing V4.1 warmup assembly: ValueError too many values to unpack.
2026-10-18 23:05:47 +08:00
minke.yu e766b3a0e0 fix(mm): _owner_span_encoder pass data_embedding_func to _move_items_to_device
Upstream refactor added the third param (defer_move check); our local
owner-span encode path kept the old 2-arg call and crashed V4.1 warmup:
TypeError: _move_items_to_device() missing 1 required positional argument
2026-10-18 22:32:28 +08:00
minke.yu e4391748e9 Merge upstream/main (6c3db0f520, 2026-10-18) into g9i-est-td
对齐上游 main 至 2026-10-18。回退基准 tag: 26.10.18。

上游官方版取代的本地 cherry-pick: #40323/#40335(Prefill CP+stream-sync)、
#40177(Decode DP-attn)、#40333->#38603(BCG 官方版)、#40178(MegaMoE 官方版)、
#42194/#41520(disagg prefill 官方重构, 删除本地 checkpoint_disagg_prefill 影子 shim)、
#42420/#42264/#41607/#42113/#41987/#40637/#41658/#41660/#42716 等。

保留的本地独有优化:
- g9i 网关集成 (sgl-model-gateway, query_cache_hit 经 TokenizerControlMixin 存活)
- hicache L3 io_aux 独立车道 + HC_TIMING 打点 (cache_controller.py)
- #38348 fail-soft hybrid prefetch 降级 (unified_radix_cache.py, 适配新
  storage_existence_cache.pool().invalidate_beyond API)
- V4.1 vision CP 放行 (deepseek_v4.py / deepseek_v4_hook.py / eager_runner.py)
- NVLink-intra host 负载修复 (cuda.bindings 指针探测 + STATE_DATA 通道)
- mooncake standalone dummy 补丁 + rank 后缀主机名
- PD debug 仪表 (_dbg_phase/_sched_idled/record_prefill_drain)
- #40429 FlashMLA smem 崩溃修复 (_FLASHMLA_SCHED_MAX_SHARED_BYTES)

#42035 无需再 revert: 上游已默认关 (SGLANG_ENABLE_DISAGG_PREFILL_CONTINUOUS_
INPUT_POLLING=False) 且硬性要求 attn_dp==1 && attn_cp==1 && 非投机。

冲突 34 文件全部解决, srt 下 1038 个变更文件 py_compile 全过。
运行级验证未做, 待 b300 构建镜像 + 冒烟。
2026-10-18 18:13:02 +08:00
Ilia Yastrebovandishandhanani 6c3db0f520 [Perf] NIXL: prepared dlists for hybrid-model state transfer (#44740)
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-18 16:59:18 +08:00
Xinyuan Tong 80fef63786 [Fix] Inkling: frame constrained output in native message tokens (#44716) 2026-10-18 16:47:56 +08:00
Huang Long 889658a5be [Fix] disaggregation: Reuse startup decode graphs across PD role switches (#43760) 2026-10-18 16:16:46 +08:00
+14 d33dd0566a [Feat] srt: Add muse spark runtime, multimodal support and tests (3/5) (#44666)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-18 01:00:24 -07:00
Shangming Cai 80914b7c9b [Refactor] sgl-router: Rename --max-in-flight to --max-inflight-requests (#44950) 2026-10-18 15:33:58 +08:00
Yuwei An 3a6e1fcf77 [Perf] DSv4.1: Bound encoder BCG metadata memory (#44963) 2026-10-18 00:33:35 -07:00
Richard WangandMick Qian de61ab53d4 [Perf] diffusion: Defer full GC collections while importing the runtime (#44606)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 15:31:10 +08:00
Zhaoyi Li f1cf2e1005 [Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index (#44412) 2026-10-18 00:29:22 -07:00
Zhaoyi Li 396e97e5dd [CI] [AMD] ci/slurm: Add a Mooncake KV-transfer test to the MI355X disaggregation nightly (#44409) 2026-10-18 00:28:34 -07:00
Zhaoyi Li 8a4239fbdc [Fix] [AMD] mooncake: Bump Mooncake to v0.3.14-rc1 for cross-node KV transfer (#44404) 2026-10-18 00:22:55 -07:00
Shuwen Wang ba3fc9441d [Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry (#43559) 2026-10-18 15:16:31 +08:00
jacky.chengandMick Qian f96a7326e6 [Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup (#43952)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 00:13:25 -07:00
Shuwen Wang a11fa06be7 [Fix] hicache: Key MLA Mamba state per TP rank in the file backend (#41907) 2026-10-18 15:10:50 +08:00
Shuwen Wang ffc4573c98 [Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend (#43816) 2026-10-18 15:09:58 +08:00
Shuwen Wang e6c2cd47a9 [Fix] hicache: Keep unified memory off the direct page_first_direct copies (#43333) 2026-10-18 15:09:26 +08:00
Liangsheng Yin df966f2345 [Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers (#44952) 2026-10-18 00:07:55 -07:00
Ziang Li 520df8a491 [Perf] weight-sync: Allocate packed receive buffers from metadata (#39264) 2026-10-18 00:04:10 -07:00
Michael 00f6cdcd1b [Fix] [AMD] kda: Reject match_cutedsl_decode on ROCm and skip its parity tests (#44089) 2026-10-18 00:01:40 -07:00
Michael 3626978a13 [Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel (#43991) 2026-10-18 00:01:11 -07:00
Bowen Wang 3671503e44 [Feat] bench: report goodput and SLO attainment in bench_serving (#43967) 2026-10-18 13:31:48 +08:00
62d4261cf4 [Doc] sgl-router: Rewrite README and add architecture guide (#43592)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-18 13:29:08 +08:00
Thomas Wang e5dc10e73b [Doc] [AMD] dsv41: Update amd cookbook 1016 (#44903) 2026-10-17 22:10:42 -07:00
Shangming CaiandShangming Cai 1ff2c1d71e [Fix] sgl-router: Fill the media fields when converting TextGenerate to Generate (#44951)
Co-authored-by: Shangming Cai <shangming.cai@radixark.ai>
2026-10-18 13:05:45 +08:00
Liangsheng Yin d20d010574 [Fix] deepseek-v4.1: Exchange late-layer DP row counts before the forward (#44948) 2026-10-17 21:46:46 -07:00
Liangsheng Yin 6fe630ae14 [CI] deepseek-v4: Set the KV pool in the BCG replay contract test (#44946) 2026-10-17 20:59:03 -07:00
Baizhou Zhang a8b939da43 Revert "[Doc] cookbook: Add Muse Spark Open SWA ratio calculator" (#44947) 2026-10-17 20:55:00 -07:00
140c57d5bf [Perf] diffusion: classify residual_gate_add operands once per call (#44478)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:51:52 +08:00
Liangsheng YinandVincent Zhong ff73d63121 [Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph (#44930)
Co-authored-by: Vincent Zhong <207368749+vincentzed@users.noreply.github.com>
2026-10-17 20:51:18 -07:00
Chunan Zeng a0b77db84d [Perf] DFLASH: build the draft num_token_non_padded scalar with torch.full (#44119) 2026-10-17 20:40:16 -07:00
Kaushik Chatterjee 3acfa2efcc [Feat] runtime: Support tokenized multimodal generation (#42833) 2026-10-18 12:35:51 +09:00
Richard Wang 01f3cba70e [Fix] dLLM: Size decode graphs to the default max running requests (#40941) 2026-10-17 20:22:39 -07:00
531b426281 [Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview (#39475) (#39475)
Co-authored-by: Kedi Wu <kediw@nvidia.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 11:15:57 +08:00
zijiexia 3b0a154e60 [Doc] cookbook: Add DeepSeek V4.1 Flash 1017 variant (#44937) 2026-10-17 20:04:41 -07:00
zijiexiaandClaude Sonnet 5.5 df24bca4ba [Doc] popular-models: Replace Kimi-K3 with Muse Spark Open (#44693)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 19:54:45 -07:00
14ad64c411 [Perf] custom_op: call register_custom_op ops directly in plain eager execution (#44158)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 10:10:10 +08:00
MickandClaude Opus 5 d748bede96 [Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention (#34385)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-18 10:03:43 +08:00
8d02b69c11 [Perf] diffusion: Replay Triton launches without the runner's per-launch hook metadata (#44471)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:58:54 +08:00
949e1a2080 [Perf] diffusion: Cache the fused linear-GELU verdict per linear (#44643)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:55:29 +08:00
f53bd436c1 [Perf] distributed: cache the eager launch of custom all-reduce v2 per size (#44642)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:54:58 +08:00
Yuwei An b234b5c32e [Refactor] TCPCG: Consolidate attention eager regions and isolate Inkling policy (4/9) (#42288) 2026-10-17 16:06:24 -07:00
Yuwei An e4512a5a17 [Perf] DSv4.1: Reduce BCG metadata memory (#44920) 2026-10-17 16:05:28 -07:00
Liangsheng Yin 563cd60345 [Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm (#44934) 2026-10-17 15:47:41 -07:00
Yi (Vincent) Zhong 05ce0e980f [Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay (#42901) 2026-10-17 14:12:00 -07:00
05c1539098 [Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker (#41343)
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2026-10-17 13:24:08 -07:00
YAMYandClaude Opus 5.5 3b34d9f8b6 [Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools (#44114)
With an FP8 KV pool on SM10x, keep the packed sparse-decode K/V scratch in FP8 instead of dequantizing to BF16 (BF16-q + FP8-KV TRTLLM-gen decode). Tests cover FP8 pool -> FP8 scratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 11:50:38 -07:00
4eb606b2ae [Feat] diffusion: support opt-in pi0.5 completion-boundary action batching (#44634)
Co-authored-by: Jiahua Huang <35223412+dadiaokua@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 20:49:44 +08:00
b15a78a881 [Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach (#40171)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 20:48:14 +08:00
23f8e6f7ec [Fix] diffusion: keep ZMQ ingress sockets on their configured port (#43732)
Co-authored-by: Sylvan Liu <syliu@cse-ai-16.amd.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 20:47:29 +08:00
+3 d4191a4624 [Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning (#44493)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
Co-authored-by: Mick Qian <mickqian@radixark.ai>
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com>
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Fabio Camaiora <fcamaiora@nvidia.com>
Co-authored-by: Linda-Stadter <lstadter@nvidia.com>
2026-10-17 20:46:13 +08:00
9771e5a4f0 [Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol (#44662)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:45:07 +08:00
MickandMick Qian e01dfd3cbc [Refactor] multimodal: Replace padding strategies with helpers (#44845)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 20:43:02 +08:00
Shuwen Wang 86e67f7d93 [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation (#43387) 2026-10-17 20:20:49 +08:00
SuperSong 025655068c [Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction (#44097) 2026-10-17 04:56:41 -07:00
SuperSong ee30dc4d40 [Fix] unified-memory: Bound joint reclaim and preserve cached prefixes (#39294)
Bound joint FULL/SWA/Mamba reclaim so feasible allocations recover capacity without evicting unnecessary cached prefixes, and impossible demands preserve the cache. Use exact unsharded paged-extend demand and update allocation regression fixtures.
2026-10-17 04:46:48 -07:00
Shuwen Wang ae28306a74 [Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay (#44220) 2026-10-17 19:13:57 +08:00
Shuwen WangandZhangheng 5fcb07da34 [Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend (#43817)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-17 11:08:17 +00:00
Woody-HuandMick 465ddf9758 [Feat] diffusion/VSA-H3: support sm80 (A100) with a raw-pointer Triton load path (#43823)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 18:47:19 +08:00
400c1dfe6e [Perf] distributed: skip the DLPack re-wrap of custom all-reduce results (#44251)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 18:44:31 +08:00
pllimax eb5739b30e [CI] pr-test-npu: Temporarily disable recommend-tests-from-coverage job (#44881) 2026-10-17 18:21:17 +08:00
Khoa PhamandClaude Fable 5.1 a126e8d640 [Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints (#44838)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-17 02:45:53 -07:00
MickandMick Qian 80f40522cd [UX] diffusion: log LoRA merge decisions only on first evaluation (#44376)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 16:47:07 +08:00
56a20d7e34 [Perf] kernels: Enter a JIT kernel's device guard only when the device differs (#44644)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:43:27 +08:00
f75232cb42 [Fix] hicache: Order write-back host copies after the forward that produced the KV (#44800)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-17 01:31:40 -07:00
0d13efb2b8 [Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend (#44767)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-17 01:30:50 -07:00
a2953d7b7f [Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay (#44850)
Co-authored-by: Pedram Ghazi <pedram.ghazi@amd.com>
Co-authored-by: David Orman <ormandj@corenode.com>
2026-10-17 01:25:33 -07:00
McZyWu 7a8804edb5 [Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP (#40516) 2026-10-17 16:19:44 +08:00
Liangsheng Yin ef26f6ad49 [Fix] deepseek-v4.1: Keep the cached KV of folded encoder replay rows (#44857) 2026-10-17 00:56:18 -07:00
Khoa PhamandClaude Fable 5.1 bce7b27aa6 [Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8 (#44842)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-17 00:19:35 -07:00
01dc08e23f [Refactor] dsv41: Shard Engram tables on hash-column boundaries (#42787)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-17 00:15:49 -07:00
Khoa PhamandClaude Fable 5.1 14c4dda24e [Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size (#44837)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-16 23:47:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 9c7b338a19 [CI] jit-kernel: Honor continue_on_error on scheduled runs (#44645)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 22:29:01 -07:00
Mohammad Miadh AngkadandMohammad Angkad 2d8a05e719 [Test] attention: Fix replicated attention test missing full_attention_interval (#44602)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 22:28:37 -07:00
Liangsheng Yin 9259553f09 [Fix] megamoe: Support EPLB rebalance on SM100 (#44847) 2026-10-16 22:22:40 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3fc08b939a [Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache (#44849)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-17 13:22:34 +08:00
Mohammad Miadh AngkadandMohammad Angkad 4fc86f73c2 [CI] glm5.2: Keep EOS in HiSparse needle test output (#44597)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-17 12:20:51 +08:00
Liangsheng Yin 4cbea9d3c7 [Doc] cookbook: Allow PD with DSpark for DeepSeek-V4.1 (#44848) 2026-10-16 21:17:32 -07:00
Xinyuan Tong 86cc2e02ec [Fix] gateway: Drop unused npyz dev-dependency to unbreak builds after zip yank (#44720) 2026-10-16 20:56:31 -07:00
Ziang Li 29a7e19de1 [Refactor] nvfp4_online: Use explicit FlashInfer 4over6 config (#41745) 2026-10-16 20:56:10 -07:00
feixiangpengandzijiexia 9590dc80cc [Doc] hicache: Explain HiCache host pool sizing with a storage backend (#44790)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-17 11:28:37 +08:00
Shuwen Wang 6a39ddf1f2 [Fix] unified_cache: Resize test allocator before allocating backed pages (#44843) 2026-10-16 20:21:49 -07:00
3603824541 [Perf] DSv4.1: Enable decoder bounded replay by default (#44403)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-16 19:52:02 -07:00
McZyWu 2d3dd3168d [Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback (#40131) 2026-10-17 10:12:41 +08:00
Khoa Pham ff979e0816 [Fix] megamoe: Respect request-pool limits in decode validation (#44816) 2026-10-16 18:56:38 -07:00
5cbe2371a5 [Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend (#44764)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 18:14:50 -07:00
Shuwen Wang 33b5d26fa6 [CI] lora: Use the canonical Llama model ID in eviction tests (#44771) 2026-10-16 18:13:37 -07:00
b62a48ee15 [Feat] sglang-server: Serve /flush_cache from the Rust frontend (#44766)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 17:24:01 -07:00
d8dd49d21d [Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker (#44785)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-16 16:55:54 -07:00
d230f2c35b [Feat] profiler: Add the forward pass id to detailed step span labels (#44775)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-16 16:51:20 -07:00
36cf028346 [Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id (#44592)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-16 16:47:50 -07:00
Khoa PhamandClaude Fable 5.1 aaf5f397a5 [Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention (#43813)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-16 15:38:33 -07:00
Yuwei An 8e3f11b219 [Perf] DSv4.1: Skip discarded KV rows during window commit (#44783) 2026-10-16 15:05:45 -07:00
a46fe83c70 [CI] tests: Fork the gloo exchange test's ranks from a preloaded forkserver (#44679)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
2026-10-16 14:53:21 -07:00
AaravandAarav d5d87899f9 [Fix] DSA: skip-topk pipeline handoff for owed outputs (#43832)
Co-authored-by: Aarav <211792885+AaravMehta-07@users.noreply.github.com>
2026-10-16 14:35:23 -07:00
Khoa PhamandTrevor Morris 71b60fa46c [Fix] engram: Avoid blocking history copies during PD admission (#44787)
Co-authored-by: Trevor Morris <tmorris@nvidia.com>
2026-10-16 14:29:05 -07:00
2948662026 [Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo (#44622)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-16 13:53:41 -07:00
metamergebotandmetamergebot f39c2693aa [CI] rust-server: Raise the ChunkEvent frame-size cap to 152 bytes (#44773)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-16 13:47:52 -07:00
Chunan Zeng 07950be745 [Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8 (#42811) 2026-10-16 13:43:18 -07:00
Mohammad Miadh AngkadandMohammad Angkad 21791e7023 [CI] [NVIDIA] DSv4.1-Flash: Split GB300 accuracy test into one file per config (#44696)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 13:40:03 -07:00
metamergebotandmetamergebot a31a8c2bbd [CI] tests: Let test_hicache_storage size its host cache to the runner (#44729)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-16 13:38:39 -07:00
Liangsheng Yin a18a908066 [Test] mem_cache: Free SWA peers below the eviction floor in the SWA repair test (#44777) 2026-10-16 12:41:39 -07:00
+14 456231e251 [Feat] kernels: Add muse spark operators and collective GEMM (2/5) (#44665)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-16 10:49:35 -07:00
ormandjandhnyls2002 80c54e6ac5 [Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA (#42716)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-16 10:45:16 -07:00
553068af25 [Fix] rust-server: Honour cache_salt and require_reasoning on /generate (#44533)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 10:38:09 -07:00
453b251dbd [Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll (#44614)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-16 10:37:11 -07:00
metamergebotandjiayisuse 989ccf4863 [Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer (#44605)
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-16 10:36:46 -07:00
63a9b7a713 [Feat] http_server: Make the multi-worker ASGI app path configurable (#44583)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-16 10:36:35 -07:00
26a2f18c02 [Feat] sglang-server: serve /v1/loads from the Rust frontend (#44532)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 10:35:37 -07:00
Xinyuan Tong a038a0dca0 [Fix] dsv41: Preserve tool-result image order and current-turn thinking (#44751) 2026-10-16 23:39:50 +08:00
Duyi-Wang ed2ebad342 [Doc] [AMD] cookbook: Refresh DeepSeek V4 AgentX PD recipe optimizations (#44454) 2026-10-16 23:34:01 +08:00
Jimmy ShongandJimmy Shong 2c2986be2b docs: add Beam 1.0 deployment cookbook (#44694)
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
2026-10-16 13:29:14 +00:00
Benjamin TruongandClaude Opus 5.5 7c99ace127 [Fix] dsa: Fix ragged seed indices before MTP draft decode (#42565)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:37:41 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 5e240a200d [Doc] cookbook: Refresh SemiAnalysis AgentX launches for Qwen3.5 and DeepSeek-V4.1-Flash GB300 (#44712)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:31:08 +08:00
Xinyi SongandHAI 4afae5568b [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance (#44657)
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-16 03:11:02 -07:00
2adc76ad61 [Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8) (#41903)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:55:25 -07:00
YC Yen-Ching Tseng 9246acd462 [CI] [AMD] aiter: pin aiter commit to 2bb549 for rocm10.1 gfx1250 (#44620) 2026-10-16 17:50:20 +08:00
kkandDuyi-Wang baf451fcd6 [Perf] [AMD] dsv4: carry aiter DSv4 GEMM tuned rows and mHC tile_m=16 until aiter ships them (7/8) (#44008)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:47:41 -07:00
f05da43d44 [Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) (#30929)
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:39:27 -07:00
42eb556e66 [Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8) (#44007)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-16 02:28:31 -07:00
caihuali95 8d229d96e0 [Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding (#42930) 2026-10-16 02:24:06 -07:00
kkandDuyi-Wang 6efc96abf9 [Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8) (#44006)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:16:06 -07:00
kkandDuyi-Wang 9fde8bcc46 [Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) (#44004)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:09:32 -07:00
kkandDuyi-Wang f0ff18353d [Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8) (#44003)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 01:58:00 -07:00
MingyiandZijie Xia 56a3c4fa16 [Doc] cookbook: Add RTX Spark Qwen3.6 NVFP4 recipe (#44707)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-10-16 01:55:41 -07:00
Liangsheng Yin 81c9d804b0 [CI] pr-check-comment: Comment on PRs when lint or title check fails (#44708) 2026-10-16 01:54:38 -07:00
kkandDuyi-Wang ab5858ac8a [Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) (#44016)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 01:50:16 -07:00
Xinyuan Tongandzijiexia 9baace8456 [Doc] cookbook: refresh MiniCPM-V 4.7 DSPARK benchmarks at the final model PR head (#44310)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-16 01:48:25 -07:00
3d61bdd2c9 [Feat] sgl-router: Serve the remaining native gRPC inference RPCs (#42681)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-16 16:23:17 +08:00
kkandwunhuang 536cdb4b04 [Feat] [AMD] docker: Install the sgl-amd-mono monokernel plugin in the ROCm image (#44621)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-16 01:20:40 -07:00
Liangsheng Yin fad7d95a7f [Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree (#44673) 2026-10-16 01:18:58 -07:00
Baizhou Zhang 06b81b120e [CI] deepgemm: Increase B200 release test timeout to 180 minutes (#44706) 2026-10-16 01:14:54 -07:00
kkandwunhuang bdc4ec15ce [Perf] [AMD] dspark: No per-step host sync in decode (1/4) (#44405)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-16 01:13:37 -07:00
ZiruandNiu Ziru 5dd37af412 [Doc] [XPU] Cookbook: Update the XPU Docker version to 0.5.22 (#44647)
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
2026-10-16 16:13:24 +08:00
jiaqiang-dot-liu a448fdf316 [Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode (#44267) 2026-10-16 00:51:55 -07:00
amote-i 9d394dfbd7 [Doc] [NPU] docs: refresh associated dependency versions in npu docs (#44695) 2026-10-16 15:48:37 +08:00
Ziang Li 7098ec387c [Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE (#44036) 2026-10-16 00:38:50 -07:00
Liangsheng YinandJingshu Chen 3a5d46e946 [Perf] pd: Stage decode-side KV preallocation lengths through pinned memory (#38955)
Co-authored-by: Jingshu Chen <96519051+elmo2019@users.noreply.github.com>
2026-10-16 00:34:56 -07:00
Xiaoyu Zhang 51faf52305 [Fix] monokernels: Honor resolved CUDA graph configuration (#44630) 2026-10-16 15:06:14 +08:00
Baizhou Zhang dfc89580f1 [Doc] cookbook: Add Muse Spark Open SWA ratio calculator (#44687) 2026-10-16 00:00:12 -07:00
amote-i 92580581da [Doc] [NPU] docs: Freeze qwen3-235b-a22b recommended image to a fixed version (#44650) 2026-10-16 14:14:05 +08:00
Mohammad Miadh AngkadandMohammad Angkad 44be30d216 [Chore] deps: Bump FlashInfer to 0.7.2 (#44591)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-15 22:51:21 -07:00
Shu Wang 83940471b4 [Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques (#44387) 2026-10-15 22:50:33 -07:00
+14 e1d20574da [Chore] docker: Add muse spark CUDA image and DeepGEMM patches (1/5) (#44664)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-15 22:46:22 -07:00
Yuwei An 06934575b9 [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage (#44131) 2026-10-15 22:32:55 -07:00
2a08e261ec [Feat] MiniCPM-V-4.7: add model support (#41993)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-16 13:24:48 +08:00
a2981ab8ea [Feat] sgl-router: Serve native gRPC ChatComplete (#42674)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:11:36 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 fd4b75c696 [Doc] cookbook: Add SemiAnalysis AgentX launches to Muse Spark Open, refresh DeepSeek-V4.1-Flash (#44654)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:45:07 +08:00
Kolade FajimiandXiaoyu Zhang 5117bfbaeb [Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100 (#44213)
Signed-off-by: kolade fajimi <fajimikolade23@gmail.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-16 12:36:02 +08:00
HuangJi 6d70787398 [Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output (#44275) 2026-10-16 12:35:04 +08:00
MickandMick Qian e5274752a8 [Doc] diffusion: align PR title conventions (#44646)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-16 11:29:20 +08:00
Mick 3420baea8c [diffusion] CI: batch performance failure-policy runner scenarios (#44508) 2026-10-16 11:04:05 +08:00
Mohammad Miadh AngkadandMohammad Angkad 6231a617d3 [Test] diffusion: Load NVFP4 block scales through the weight loader in tests (#44595)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 10:56:27 +08:00
ashwini rathiandClaude Opus 5.5 75e25a21bd [Fix] [XPU] nightly: Fix rerank, zimage, flux2 and EAGLE parity (#44100)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:27:51 +08:00
Chetan Kumar VermaandClaude Sonnet 5 75e77fca25 [Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions (#43768)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-16 10:22:09 +08:00
Shuwen Wang f9df33494a [Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held (#44386) 2026-10-15 19:17:51 -07:00
Chetan Kumar VermaandClaude Opus 5.5 1e33d2c4bd [Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all (#39989)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 10:08:50 +08:00
4e85e8ed37 [Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it (#44430)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: yeqcharlotte <yeqcharlotte@users.noreply.github.com>
2026-10-15 18:52:33 -07:00
shodocoandmetamergebot 2957ff3165 [Fix] sglang-server: Keep the matched stop token in output_ids (#42637)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-15 18:51:47 -07:00
sglang-botandsglang-bot c15edf314d [Doc] blogs: Sync LMSYS SGLang blog cards (#44616)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-16 00:52:14 +00:00
e3961eb6fa [Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler (#44440)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com>
2026-10-15 17:50:43 -07:00
f110b2f801 [Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info (#44531)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 17:50:27 -07:00
d393298736 [Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams (#44537)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 17:49:49 -07:00
metamergebotandYe Qi bda6ef7e01 [Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing (#44568)
Co-authored-by: Ye (Charlotte) Qi <7269575+yeqcharlotte@users.noreply.github.com>
2026-10-15 17:48:50 -07:00
ed8c3139f1 [Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends (#41303)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-15 17:32:50 -07:00
Khoa Pham a93f16f0cf [Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV (#43922) 2026-10-15 17:21:43 -07:00
Liangsheng Yin d14fa5db9b [Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs (#44598) 2026-10-15 17:20:02 -07:00
Khoa Pham 4cca7fcd3e chore: bump sgl-kernel version to 0.4.10 (#44590) 2026-10-15 15:50:51 -07:00
ec7ab9d0c2 [Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins (#44353)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-15 15:50:25 -07:00
0b1a73599a [Feat] rust-server: Apply the x-override-* request headers (#44312)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 15:50:04 -07:00
William Hu 4e0609ea27 [Fix] moe: Preserve negative routing sentinel in EP mapping (#40498) 2026-10-15 15:05:08 -07:00
Yuwei An 91f6ef2322 [DSv4.1] Fuse request-window KV copies (#44555) 2026-10-15 14:11:51 -07:00
Xinyuan Tong 877be2974a [CI] pr-test: Let stage wait jobs reach their max-wait budget (#44410) 2026-10-15 13:34:49 -07:00
snadampal aa5c7e887c [Feat] docker: Add AWS EFA image variants built on published images (#41006)
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
2026-10-15 13:19:22 -07:00
Ziyi XuandClaude Opus 5.5 bc76a6b819 [DSv4] Sparse prefill: build compressed lengths without a host sync (#44557)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:45:56 -07:00
87211088c9 perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell (#36810)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-15 12:09:09 -07:00
02a43eeef6 [Feat] rust-server: Attach configured metadata headers to a healthy /health_generate (#44315)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 12:04:23 -07:00
173e0f2e9b [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode (#44173)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-15 12:02:59 -07:00
Michael bba535ebb4 [Fix] [AMD] docker: Repair the AITER build step after the 2bb549102 pin bump (#44534) 2026-10-15 11:18:41 -07:00
c5a5bf0d0b [Fix] pp: Order output send/recv by rank parity to fix cross-node hang (#44453)
Co-authored-by: chuixue <chuixue@minimaxi.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-16 01:46:37 +08:00
Shangming Cai c7fb57d990 [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349 (#44509) 2026-10-16 01:28:57 +08:00
a8aaf95b58 [Cookbook] Add cookbook for muse spark open (#44527)
Co-authored-by: thomawan <thomawan@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
Co-authored-by: Thomas Wang <1am9trash@gmail.com>
Co-authored-by: zijie <zijie.chen@amd.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-15 10:20:38 -07:00
Aleksi Vesantoandjacky.cheng 5ef8319f16 [Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation (#27702)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-16 00:40:41 +08:00
Yihao WangandClaude Opus 5.5 5b12ec163c [Doc] cookbook: Expand the FLUX.3 Image Klein model description (#44515)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 08:56:54 -07:00
Jimmy ShongandJimmy Shong 05cbe241be [Doc] cookbook: Add Nemotron 3.5 Super VL (#44510)
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
2026-10-15 08:39:14 -07:00
28b98e5d13 [Doc] cookbook: Add the FLUX.3 prompt upsampler section (#44512)
Co-authored-by: Sandro Cavallari <scavallari@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 08:38:33 -07:00
YC Yen-Ching Tseng e87e32b311 [Feat] [AMD] docker: Release ROCm 10.1 for gfx1250 in parallel with ROCm 10.0 (#42699) 2026-10-15 22:59:54 +08:00
a57707377f [Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware (#44072)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 22:42:33 +08:00
qiaonan02 c4870d98cf [Fix] glm5_next: Correct num_experts_per_tok default to 8 (#44093) 2026-10-15 21:53:31 +08:00
09b7383624 [diffusion] model: support FLUX.3 image (#44418)
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
Co-authored-by: Mick Qian <mickqian@radixark.ai>
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com>
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 21:50:59 +08:00
elvischenvandClaude Opus 5.5 6fe5596ebb Support EPLB for Nemotron-H (#43685)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:50:09 +00:00
ormandjandBenjamin Oldenburg c5bf3aafbc Fix overflow in legacy and v1 DSA top-k selectors (#37625)
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.co.th>
2026-10-15 21:44:58 +08:00
jiaqiang-dot-liuandCursor 2f5407da28 [Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path (#44266)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-15 21:22:46 +08:00
sogalin_codegen 2092160d42 [AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 (#41988) 2026-10-15 21:08:44 +08:00
amote-i abf92826ee [NPU] [DOC] update NPU best practices and model deployment docs to CANN 9.1.0 (#44432) 2026-10-15 20:22:54 +08:00
Zhang Yiyang (SII)andMick Qian 23c2a026db [diffusion] model: support MOSS-Image (#44325)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 19:23:56 +08:00
Cheng Wan f7ad4345b8 [Refactor] deepseek-v4: Use explicit stage boundaries and row transport (#43607) 2026-10-15 02:05:28 -07:00
Cheng Wan 3a835cf417 [Refactor] inkling: Use short-convolution stage boundaries (#43606) 2026-10-15 02:04:46 -07:00
Cheng Wan e1f9fce7b3 [Refactor] layer-boundary: Bind residual completion to producer updates (#43605) 2026-10-15 02:04:02 -07:00
Cheng Wan d59c3448b0 [Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling (#43604) 2026-10-15 02:03:27 -07:00
Eddie Zhou 71a17dbc24 [AMD][CI] setup: Skip redundant pip-cache chown walk and MORI rebuild (#42148) 2026-10-15 16:21:46 +08:00
93dbe88183 Fuse MiMo verification RoPE and KV cache writes (#43106)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 16:15:24 +08:00
kangwangamd 28703575f9 [AMD] Update ROCm AITER pin to 2bb549102 (#44379) 2026-10-15 16:10:41 +08:00
GeorgeSherifandXiaoyu Zhang 8f853b0041 [Doc] cookbook: Add Jais 3 deployment recipes (#42766)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 16:09:44 +08:00
86f4ede4fb [Fix] dsa: Make the fused top-k exact on an overflowing threshold bin (#37941)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: DarkSharpness <2040703891@qq.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 15:52:48 +08:00
9ea5472833 [HiCache] Add the page_unified host KV layout and the unified L3 key (#39727)
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-15 15:46:13 +08:00
33618d4f00 [Feat] runtime: Include FlashInfer’s default tactic in startup tuning. (#43927)
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:30:45 +08:00
Liangsheng Yin a776851c97 [Refactor] scheduler: Check the prefill budget at the admission loop head; drop a redundant PD gate (#44420) 2026-10-15 00:16:56 -07:00
fa66e7d816 [NPU] [Bugfix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism (#29753)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
2026-10-15 09:58:47 +03:00
ashwini rathiandClaude Opus 5.5 01ac2cf36d [CI] [XPU] nightly: Split Intel Main and Wheel workflows (#44108)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:56:27 +08:00
MickandMick Qian f2b73d4f9b [diffusion] doc: document reference resizing, LoRA auto mode and image profiling (#44377)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 13:41:44 +08:00
Dayuxiaoshui a50c17e979 [diffusion] perf: fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel (#38650) 2026-10-15 13:39:27 +08:00
676c038aa5 [diffusion] fix: fix the load duration missing from subclassed server tests (#40657)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-15 13:38:23 +08:00
Liangsheng Yin f9bdb74a5a [Fix] scheduler: Count a request's held req_to_token row once at admission (#44401) 2026-10-14 22:29:10 -07:00
Yuxingwang-intel 93842f52fe [CPU]: Add group-aware CPU SHM collective kernels (#35435) 2026-10-15 13:22:20 +08:00
Michael 4dc9b5c68c [Test] [AMD] diffusion: Expect CUDA-only fast paths to be off on ROCm in kernel tests (#44311) 2026-10-15 13:18:01 +08:00
1d809bbbbd [Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails (#43502)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-10-14 22:04:58 -07:00
kkandroot e573d91275 [CI] tests: Fix stale fakes in the CP strategy and request-class override unit tests (#44380)
Co-authored-by: root <root@crsuse2-m2m-223.us-east2-a.compute.internal>
2026-10-14 21:35:20 -07:00
Khoa Pham d4daf52369 [Fix] dsv41: Bound MoE routing to live SWA graph-tail rows (#44347) 2026-10-14 21:14:28 -07:00
AMD-yanfeiwang 08f15e76bb [Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware (#43511) 2026-10-15 12:03:01 +08:00
WenhaoZhangandMick c92108bb93 [diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight (#44058)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-15 11:59:03 +08:00
Liangsheng Yin 3b5ae4ea86 [Fix] scheduler: Drop finished requests from the running batch before every admission (#44351) 2026-10-14 20:40:05 -07:00
2528cd2b91 [PD] Send DSA indexer state only past decode_prefix_len (#44140)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-15 11:25:18 +08:00
Avaya Aggarwalandniehen6174 0cf683ea8d [diffusion] comfyui: return several server-mode images as one IMAGE batch (#44088)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-15 10:38:37 +08:00
Thomas Wang 4851d182a1 [Doc][AMD] Update v41 amd cookbook 1015 (#44368) 2026-10-14 19:33:08 -07:00
xiaofei-zheng a12814c158 [Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk (#43643) 2026-10-15 09:36:16 +08:00
Yuhao YangandClaude Opus 5.5 9fa9308e95 [Doc] cookbook: Add Step 5 Preview cookbook (#44357)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 09:21:06 +08:00
9d68ecce2e [DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits (#44051)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 09:00:19 +08:00
xiaofei-zheng c5c4bbbd25 [Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP (#43642) 2026-10-15 08:59:34 +08:00
Michaelandjacky.cheng 8333ccfccb [Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph (#43770)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-14 17:40:07 -07:00
Yuwei An 64a0a08f0a [DSv4.1] Support TRT-LLM decoder bounded replay with BCG (#44326) 2026-10-14 17:39:52 -07:00
YAMYandYuwei An c891c5c7ca [DSv4.1] Fold encoder-SWA replay into the extend forward (#43166)
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-10-14 17:27:07 -07:00
Shuwen Wang 3c28614f12 [Fix] hicache: Defer query beliefs until the rank-synchronized hit drain (#43108) 2026-10-14 17:25:08 -07:00
Aditya Singhandhnyls2002 a6a8c56015 [Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config (#33715)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 17:20:21 -07:00
0536cec73d [Feat] metrics: Expose multi-detokenizer router CPU usage (#43323)
Co-authored-by: raghul chandramouli <raghul@c1xind0180-0243.local>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 17:17:22 -07:00
Junnan Liandhnyls2002 0f36c8015c [Bug] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields (#31892)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 16:56:56 -07:00
209f8220d6 [Fix] sglang-server: Honour extra_key on the Rust frontend's /generate (#44207)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:44:31 -07:00
e7d6a88bfc [HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows (#43932)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: houseroad <houseroad@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-14 16:43:24 -07:00
aa8793f33b [Rust] Serve scheduler metrics on the Rust frontend's /metrics (#43942)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:43:06 -07:00
2a9f6a25a7 [Core] Add subclass hooks for the generation request class and optimistic prefill eligibility (#44113)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: CuiCoco <CuiCoco@users.noreply.github.com>
2026-10-14 16:42:30 -07:00
5f2861bf59 [Perf] session: Cache multimodal prompt token counts across turns (#44112)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-14 16:41:34 -07:00
9732188a58 [Rust] Serve /weights_checker from the Rust frontend (#43926)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:41:10 -07:00
FREETRUMPandhnyls2002 97d3150a6a [Perf] Cache msgspec struct fields in the batch splitter (#43520)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 14:51:26 -07:00
4273814771 [Perf] detokenizer: Group return IPC sends by tokenizer worker (#44288)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-14 14:26:51 -07:00
Charles Chenandalphabetc1 d3dc14eb38 [mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache (#40857)
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-10-14 14:19:51 -07:00
Rain Jiang 12dc623102 [Chore] rust: Remove the sglang-renderer crate (#43451) 2026-10-14 11:43:37 -07:00
Shuwen Wang b67785153a [Fix] glm5.2: Align HiSparse warmup backends and move test to nightly (#44303) 2026-10-15 01:28:03 +08:00
Yoray Zack 87ed1b9526 [4/N] elastic-ep: Scale at DP-attention replica granularity (#33728) 2026-10-15 01:17:01 +08:00
c964d9ba20 [Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys (#42818)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-15 00:50:19 +08:00
amote-i 3a9cae27ed [Doc] [NPU] docs: Update NPU CANN version to 9.1.0 (#44177) 2026-10-15 00:27:18 +08:00
Michael 1737251730 [Test] [AMD] diffusion: Run the IPC all-to-all transport tests only on CUDA (#44096) 2026-10-14 23:04:23 +08:00
010776e94f [KV events] Publish opaque LoRA block IDs and adapter names (#42817)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 22:56:28 +08:00
Xinyuan Tong ed40c7fc66 [Doc] cookbook: add Ling-3.1-flash (#44301) 2026-10-14 22:40:01 +08:00
Kan WuandClaude Opus 5.5 d2a436446c [Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models (#44252)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:21:01 +08:00
204644c2dc [Fix] sglang-processor: Follow Python's OpenAI layer through #43027, #37577, #43006, #43026, #43005 (#44243)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 21:06:13 +08:00
triple-mu 5e90312eb1 [diffusion] perf: support bit-exact norm/bias fusions and channels_last decode for Qwen-Image 2.1(#41340) 2026-10-14 20:47:46 +08:00
Shuwen Wang 7b53a13401 [Fix] glm5.2: Align PD HiSparse prefill KV layout (#44256) 2026-10-14 20:40:28 +08:00
bab9c81fea fix: FA4 paged-KV zero-size page-table registers on SM90 (#35757)
Co-authored-by: razorback16 <razorback16@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-14 19:10:15 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 3b72b70ef4 [CI] vlm: Pin baidu/Unlimited-OCR revision in test_unlimited_ocr_server (#44235)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:50:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad c82e8bd5fe [Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode (#44210)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-14 17:31:19 +08:00
df1a089cd3 [Spec] Fix DFLASH verify mask policy for wrapped attention backends (#43054)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-14 17:15:36 +08:00
817781b828 [Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor (#43689)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 17:09:35 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 72730d6360 [Doc] cookbook: Mirror SA cluster NCCL_CUMEM_ENABLE in AgentX launches, fix GPU counts (#44228)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:07:40 +08:00
c54ba66cd4 [Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo (#43639)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 15:39:44 +08:00
Mohammad Miadh AngkadandMohammad Angkad 6daae9c98e [Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted (#44023)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-14 00:01:32 -07:00
58f2c923ed [Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers (#44218)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
2026-10-14 14:59:06 +08:00
Kan WuandClaude Opus 5.5 53a24c448e [Feat] sgl-router: Render Kimi-K3 through sglang-processor (#43608)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:40:18 -07:00
Michael 6f646563a2 [AMD] diffusion: expect the residual-gate fast path to be off on ROCm in test_modulate (#40660) 2026-10-13 23:34:31 -07:00
Ma Mingfei 68c18a5701 Revert "[XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend" (#44216) 2026-10-14 14:27:46 +08:00
desmond-intel b87d62e581 [XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend (#39805) 2026-10-14 14:24:04 +08:00
Shunkangz 0d130f1915 [Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8 (#39724) 2026-10-13 23:19:46 -07:00
7ca4cbfcef [Disagg] Return prompt routed experts from the PD prefill server (#44002)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-13 23:13:44 -07:00
AndyLi429andAndyLi429 05bb5afb62 [NPU][Bugfix] use CANN libdevice for DSpark Markov sampling (#43849)
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com>
2026-10-14 14:12:22 +08:00
Zhaoyi Li e29eee18df [AMD][DI][CI] nightly mi355x disagg: stage packed logs under RUNNER_TEMP, not /tmp (#43945) 2026-10-14 13:53:40 +08:00
Kan WuandClaude Opus 5.5 4560da58ee [sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor (#43625)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:19:44 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 d5f80aa3c3 [Test] dsv4: Fix Hopper indexer replay test after DeepSelect decode (#41561) (#44193)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:15:35 +08:00
ANSHUMAN TRIPATHY 30bdd09aed DSPARK enable for XPU (#33739) 2026-10-14 13:10:24 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 721953c6f6 [Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main (GLM-5.2 NVFP4 B300 agg TP4, Dynamo frontend counts) (#44065)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:03:51 +08:00
Khoa Pham e00754bab1 [DSV4.1] Derive the late-layer DP geometry from the captured shape under the breakable prefill graph (#44144) 2026-10-13 21:54:35 -07:00
Mohammad Miadh AngkadandMohammad Angkad ff8d964cf5 [CI] glm5.2: Temporarily disable PD HiSparse+MTP B300 test (#44192)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 21:53:58 -07:00
chenxb002andchenxiaobing 491d4ef178 [MLU][CI] Bootstrap MLU runner workflows (#41005)
Co-authored-by: chenxiaobing <chenxiaobing@cambricon.com>
2026-10-13 21:40:49 -07:00
Chetan Kumar VermaandClaude Opus 5.5 2c1e85964e [diffusion] cache-dit: respect CFG parallel for dual-transformer models (#43704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 12:25:07 +08:00
xinchen9andMa Mingfei 2428d0f531 [XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba… (#42122)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-10-14 12:10:12 +08:00
16176278d0 [diffusion] perf: replay small Triton wrappers' launches per operand signature (#43859)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:18:19 +08:00
Mohammad Miadh AngkadandMohammad Angkad 4f751da975 [Agents] Add a rule to prefer sgl-eval for accuracy evals (#44040)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 20:17:04 -07:00
53fe21cdfd [diffusion] perf: replace padded rows without a host sync for Z-Image (#44069)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:15:28 +08:00
xiaofei-zhengandXiaoyu Zhang 7e226b1b71 [AMD] Use the preallocated EAGLE topk=1 draft chain on HIP (#43640)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 11:06:07 +08:00
5efb7384c1 [diffusion] comfyui: consolidate 6 unrelated plugin fixes (#43493)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-14 10:56:09 +08:00
Cheng Wan c053fc71d3 [CI] pr-title: Validate pull request title format (#44165) 2026-10-13 19:53:52 -07:00
Ziyi XuandClaude Opus 5.5 4f5fcbfef5 [JIT] Clean up kernel headers: move runtime helpers, add once.h (#41147)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 10:46:51 +08:00
yuyu5333andClaude Code b50fdeb0ef [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. (#41561)
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-14 10:46:12 +08:00
Ma MingfeiandMingxuZh 87beb3a052 [CPU] Temporarily disable subblock sparse attention test on Xeon CI (#43468)
Co-authored-by: MingxuZh <109504044+MingxuZh@users.noreply.github.com>
2026-10-14 10:44:24 +08:00
13a6dee99b [Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16 (#42837)
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 10:32:50 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7f0434e4af [GLM-5.3-Flash] Drop the NextN layer from the KDA state pool (#44098)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 19:31:02 -07:00
Avaya Aggarwal ac187c0cd5 [diffusion] Bound frame_interpolation_exp to RIFE's supported range (#43014)
Signed-off-by: onepunchmonk <aggarwal.avaya@yahoo.com>
2026-10-14 10:29:28 +08:00
JinandXinyuan Tong 44b44774c3 [Bugfix] Keep rope_theta when --json-model-override-args overrides rope_scaling (#43067)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-14 10:16:35 +08:00
Baizhou Zhang 536d065370 [CP] Automatically enable BCG for supported prefill CP configurations (#44022) 2026-10-13 19:00:42 -07:00
shyeh25andClaude Opus 5.5 684fea8af3 [DSV4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches (#43993)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:36:15 -07:00
Liangsheng Yin 2ebdf9360b [Fix] Work around a CPython fork deadlock in the CPU preprocess pool (#44143) 2026-10-13 16:31:57 -07:00
b3991b53da GLM5.2 DSA Indexer Fallback (#32947)
Co-authored-by: Shu Wang <shuw@login-hecate03.hecate.clusters.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 16:27:33 -07:00
Khoa Pham 5917604a95 [DSV4.1] Support decoder SWA bounded replay with CUDA DP attention (#43812) 2026-10-13 16:10:57 -07:00
HEJIAN SANGandClaude Opus 5.5 a832609b92 [Feature] Return output top logprobs as flat base64 arrays on request (#42890)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:58:31 -07:00
baa632b049 [Score] Send stacked GPU embedding overrides through a CUDA IPC pool (#39730)
Co-authored-by: Sundara Raman Ramachandran <sundar24295@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-13 15:46:50 -07:00
Andrew Briand 08550acc28 Mark FP32 redux asm volatile (#44137) 2026-10-13 15:08:52 -07:00
Jiaqi LiandZijie Xia 5569b14d26 fix(benchmark): request token IDs when counting OpenAI prompts (#34511)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-10-13 14:31:45 -07:00
Yizheng Jiao 2341e647d3 [DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph (#43736) 2026-10-13 13:55:09 -07:00
79af3df346 [gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request (#42673)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:46:15 -07:00
huangtingwei 18210ed849 [HiSparse] Add speculative decoding support with multi-step KV cache swapping (#37771) 2026-10-13 13:38:31 -07:00
Xinyuan TongandNihar Patel 7dc2930585 [Feature] Add Aaronson-Gumbel text watermarking (#37577)
Co-authored-by: Nihar Patel <nihpat95@gmail.com>
2026-10-13 12:47:50 -07:00
Xinyuan Tong 5eda6b7e67 docs: add MiniCPM-V 4.7 cookbook (#43829) 2026-10-13 12:04:47 -07:00
ce5352a814 [Rust] Serve any model name on the OpenAI chat and completions routes (#43923)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:58:57 -07:00
01f3ed18a5 [unified memory pool] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode (#43919)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-13 11:58:24 -07:00
7e3da1b784 [HiCache] Extract the buffer-mode prefetch and load-back lifecycle into pipeline operations (#43870)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:51:44 -07:00
5bcaf0f8ed [Core] Add request-level customized_info keys and mark Full prefill capture batches (#43772)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-13 11:45:52 -07:00
149ef7d1b7 [Rust] Let an embedding package wrap the frontend HTTP app (#43924)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:44:43 -07:00
a8381d73be [HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction (#43868)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:39:48 -07:00
Khoa Phamandvincentzed 8bf145d119 [DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP (#43735)
Co-authored-by: vincentzed <207368749+vincentzed@users.noreply.github.com>
2026-10-13 11:35:34 -07:00
6ade23696d [Core] Keep request priority within the signed 64-bit range in radix caches (#43920)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: qandrew <qandrew@users.noreply.github.com>
2026-10-13 11:33:26 -07:00
7f9f742535 [Rust] Refuse an unsupported multimodal model before the Rust listener binds (#43925)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:30:43 -07:00
2a8e9db3b0 [lora] Support LoRA for the NemotronH_Nano_VL_V2 model family (#39798)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-10-13 11:29:48 -07:00
ff3f8c348c [Observability] Report per-request storage prefetch and staging waits in meta_info (#43946)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:13:01 -07:00
Rain Jiang e98c6d7b25 Rainj me/rust bench (#43950) 2026-10-13 11:07:42 -07:00
Kailash RanganathanandYanbin Jiang c0584cf9af [perf] [multi-lora] remove empty CTAs from fused LoRA expand and add split-K to shrink (#41951)
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-10-13 10:16:14 -07:00
Zhaoyi Li e9400c5a8f [AMD][DI][CI] kimik26 2p1d-ep16: capture CUDA graph on decode (eager decode is slow) (#43985) 2026-10-13 08:46:53 -07:00
Zhaoyi Li 96f519e4fa [AMD][DI][CI] glm52 2p1d-ep16: capture CUDA graph on decode (eager decode is too slow) (#43977) 2026-10-13 08:45:46 -07:00
11cd2701a8 fix(openai): preserve zero cached tokens when cache reporting is enabled (#43005)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 23:29:19 +08:00
3f1e185f6e fix(openai): keep chat stream created timestamp stable (#43026)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 22:43:33 +08:00
jiaqiang-dot-liuandCursor 4c5ed0597e Skip fully-masked KV tiles in sliding-window extend attention (#39769)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-13 22:40:53 +08:00
27b68303d0 fix(openai): omit absent tool_calls from chat responses (#43027)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-13 20:56:58 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 8e37c6cf37 [CI] Run test_pi05_modelopt_fp8_cuda in the kernel-unit lane (#44055)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:39:02 +08:00
Ho-Ren (Jack) ChuangandXiaoyu Zhang 5100f8d11d [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache (#41790)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-13 20:30:13 +08:00
amote-i 29974d543e [NPU] [DOC] update kimi k3 cookbook and supported features on NPU (#44052) 2026-10-13 20:27:26 +08:00
jlqibmandClaude Opus 5 74e0a47863 Fix/compressed tensors nvfp4 hopper (#40022)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-13 20:10:31 +08:00
LittleNyimaandMick Qian 5727f08932 [diffusion] feat: add configurable reference image resizing for MiniMax H3 Ref2VA (#35839)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 19:48:18 +08:00
WenhaoZhang 85254b5a8f [diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode (#43581) 2026-10-13 19:47:02 +08:00
Yunhao Zhouand周云昊 991a29d0ad fix(multimodal): convert grayscale tensors to RGB in PIL backend (#42127)
Co-authored-by: 周云昊 <zhouyunhao@zhouyunhaodeMacBook-Pro.local>
2026-10-13 19:03:48 +08:00
4b5b727145 Fix EVS for NemotronH_Omni_Reasoning_V3 (#42933)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 10:59:00 +00:00
MichaelandClaude Opus 5.5 aa15b06abe [AMD][CI] Run test_tracing.py in the extra tier, matching CUDA (#43928)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 02:45:58 -07:00
Xiaoyu Zhang 4e0fa52530 [Perf] Compact GDN causal-conv grids for skewed prefill batches (#43570) 2026-10-13 17:45:06 +08:00
Ziang Li 01f6840693 [FlashInfer] Add startup autotuning for MegaMoE (#43789) 2026-10-13 02:30:31 -07:00
Mohammad Miadh AngkadandMohammad Angkad 195cb7afb4 [CI] Use sgl-eval for GLM-5.3-Flash GSM8K tests (#43975)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 02:26:50 -07:00
Shu WangandAnish Shanbhag e0c957e222 [HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation (#41769)
Co-authored-by: Anish Shanbhag <ashanbhag@nvidia.com>
2026-10-13 02:07:54 -07:00
Shu WangandPo-Han Huang 27be99ae73 [NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe (#40772)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 02:00:24 -07:00
Chunan ZengandClaude Opus 5.5 7ff3a11284 [Perf] Enable fast-math for JIT activations on Blackwell (#43236)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:18:05 +08:00
AMD-yanfeiwang d8f8522348 [router] power_of_two: in-flight request load for --dp-aware decode (#43483) 2026-10-13 14:50:54 +08:00
Michal ShalevandPo-Han Huang 35a014c7e8 [PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids (#43723)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-12 23:48:13 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 dbad07a06f [Docs] Cookbook: refresh SemiAnalysis AgentX launches (DeepSeek-V4.1 GB200/GB300, GLM-5.2 FP8 B300) (#43983)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:39:45 +08:00
50a144057c [AMD][PD] enable mori transfer for PD decode DCP (#37358)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-13 14:35:09 +08:00
Xiaoyu Zhang 46b7357779 [Diffusion] Speed up LTX-2 guidance and video output (#43667) 2026-10-13 14:23:33 +08:00
Jimmy Shong 3eebf987ed [PD] Support heterogeneous TP for hybrid SWA KV transfer (#40909) 2026-10-12 23:21:54 -07:00
603a05e6c0 [ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP (#43328)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-12 23:17:49 -07:00
AMD-yanfeiwang 17c3b259f5 [router] Fix power_of_two with --dp-aware (#42027) 2026-10-13 14:15:36 +08:00
Zhaoyi Li fc404bbbb3 [AMD][DI][CI] dsv4pro-fp8 2p1d-ep16-mtp: raise prefill mem fraction for MORI EP v2 memory (#43944) 2026-10-12 23:03:57 -07:00
Xinyi Song a69e893b1a [AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1 (#42911) 2026-10-12 22:33:53 -07:00
jacky.chengandMick Qian 42e43cf33f [AMD] Fix Flux2 kv-gather abort on non-contiguous K/V (#43793)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 22:20:55 -07:00
kkandwunhuang 43356304b4 [AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels (#43968)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-12 22:09:55 -07:00
WenhaoZhangandMick Qian ea13359da9 [diffusion] perf: implement paint_use_remesh (decimate to 40k faces before UV unwrap) for Hunyuan3D (#42770)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 12:28:32 +08:00
6a89181614 tokenizer perf optimization (#39771)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-13 12:27:21 +08:00
WenhaoZhangandMick Qian feba04fa49 [diffusion] fix: avoid redundant LoRA scaling while preserving exact numerics (#43028)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 12:27:04 +08:00
WenhaoZhang cda267ab0f [diffusion] fix: keep MiniMax-H3 VAE fused norms off ROCm (#43350) 2026-10-13 12:26:16 +08:00
WenhaoZhang b4822bae03 [diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode (#43504) 2026-10-13 12:01:48 +08:00
4d5692b6cd fix(openai): honor tool_choice=none in DeepSeek encoders (#43006)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 11:41:50 +08:00
Piotr MazurekandCodex 40f5d9411f Reuse Hugging Face LFM2-MoE configuration (#43939)
Co-authored-by: Codex <codex@openai.com>
2026-10-13 11:29:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7bd41aec12 [CI] Remove Mistral Small 4 e2e test (#43973)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-12 20:14:22 -07:00
Thomas Wang c2dfd5d3d6 [AMD] Update v4 amd cookbook 1012 (#43962) 2026-10-12 19:57:55 -07:00
Xiaoyu Zhang 03416e8c0f [Fix] Account for captured inputs in DeepGEMM masked workspace budget (#43563) 2026-10-13 10:53:10 +08:00
Liangsheng Yin 43e4e84731 [PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode (#43936) 2026-10-12 19:32:03 -07:00
Yuwei An 07566cbf0d [DSv4.1] Temporarily disable B300 PD/CP accuracy test (#43948) 2026-10-12 19:31:10 -07:00
75a0d91c1f [Fix] Preserve HTTP errors for non-streaming /v1/messages (#43796)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 10:10:45 +08:00
45418d7dc6 [HiCache] Avoid full-store scans for uncached file prefix lookups (#42077)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-12 18:52:50 -07:00
Yuwei An 23150f7ae0 [DSv4.1] Reduce sparse-prefill workspace padding (#43921) 2026-10-12 18:51:36 -07:00
Ziyi XuandClaude Opus 5.5 107cff72d3 [DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows (#43833)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:45:16 -07:00
e775fa7253 [Build][CI] Drop Python 3.10: require Python 3.11+ and run CUDA CI on Python 3.12 (#43021)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-10-12 18:40:16 -07:00
YAMY 59445b3da1 [DSV4][CP] Write low-ratio page indices when CP prefill runs paged attention (#43935) 2026-10-12 18:36:11 -07:00
Michael 7224fce1b5 [AMD] Recalibrate test_quark_mxfp4 GSM8K thresholds to observed MI355X noise (#41980) 2026-10-12 16:46:03 -07:00
Yizheng Jiao 6a7abcfc9b [DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD (#43734) 2026-10-12 15:32:17 -07:00
Khoa Pham 93809202a3 [Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs (#43918) 2026-10-12 14:50:04 -07:00
Duyi-Wang 6781da8e5b [AMD] [Docker] Update MoRI to v1.2.4 (#43834) 2026-10-12 14:45:42 -07:00
Thomas Wang 64cffed139 [AMD] Fix test_compact_attention_hip.py CI (#43836) 2026-10-12 14:43:19 -07:00
Thomas Wang d6853f5066 [AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill (#43803) 2026-10-12 14:42:02 -07:00
Michael 6c2a682b59 [AMD] Keep the FP4 varctx schedule test off AITER's 2048-row compile (#41793) 2026-10-12 14:39:08 -07:00
Zhang, JiejingandJohnQinAMD 6966c5d027 [ROCm] Eliminate HiCache migration stalls and use exact-size host allocations (#42553)
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: JohnQinAMD <yanyuan.qin@amd.com>
2026-10-12 14:05:51 -07:00
ix-48 b60444c621 Fix autograd retention in CUDA graph capture tail hooks (#43854) 2026-10-12 12:31:18 -07:00
Liangsheng Yin 0efbb3d2b9 [DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept (#43791) 2026-10-12 11:35:57 -07:00
Liangsheng Yin 392039d752 [DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager (#43790) 2026-10-12 11:35:41 -07:00
Shuwen Wang f92199b569 [Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler (#40896) 2026-10-13 00:03:21 +08:00
Sam ShleiferandClaude Fable 5 d88d5c325a Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init (#37525)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-12 23:27:21 +08:00
Yuan Luoandluoyuan.luo 23a1f53fc3 [QSA] Bound the decode indexer's work by the context length (#43463)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-10-12 22:47:45 +08:00
AlexandrandАртем Савкин 1a553927b0 [CI][Diffusion] Add caching for npu GT images in CI (#37426)
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
2026-10-12 17:18:01 +03:00
279f1fad90 [diffusion] chore: bind worker CPUs to the GPU's NUMA node when memory policy is not permitted (#43756)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:09:41 +08:00
Артем Савкин d1357bf87b [NPU] Diffusion CI fixes (#43210) 2026-10-12 14:53:51 +03:00
ming_wang c6e822c29b [Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change (#43309) 2026-10-12 19:24:05 +08:00
Sasha SidorovandShangming Cai 7953751bfa [PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution (#41783)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-12 18:49:52 +08:00
8afe107c24 [XPU] Upgrade XPU backend to PyTorch 2.14 (#42003)
Co-authored-by: Ashwini Rathi <arathi@habana.ai>
Co-authored-by: Y <ashwini.rathi@intel.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-10-12 18:24:48 +08:00
CandCheng Wan 28621739fc [Fix] Make tokenizer request-state initialization atomic (#41911)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-12 03:10:57 -07:00
180933518e [sgl-router] Serve the Anthropic Messages API on /v1/messages (#42316)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-12 17:56:08 +08:00
Jinyan Yi f006fb49ec [NPU] Stabilize HiCache and registered CI tests (#43308) 2026-10-12 17:10:26 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 b0c84d4518 [Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) (#43799)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:08:11 +08:00
Baizhou Zhang 450837e770 [Docs] Update GLM-5.3-Flash playground: B300 default, CP8, EP and PD (#43802) 2026-10-12 01:38:12 -07:00
BingjiaWang 62ff466ac0 [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (#40323) 2026-10-12 01:05:21 -07:00
BingjiaWang 54409c2fa0 [BCG][DSV4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs (#38603) 2026-10-12 01:03:10 -07:00
Anupa SajikumarandSiju Samuel 8d491fbc3e [XPU] Support decode context parallelism (DCP) on Intel XPU (#34355)
Co-authored-by: Siju Samuel <siju.samuel@intel.com>
2026-10-12 15:50:33 +08:00
Ke Bao a70a83bcd2 Keep NIXL Mamba checkpoint keys separate by TP rank (#43594) 2026-10-12 15:40:59 +08:00
Ke Bao b6ba148384 Capture KV host transfer buffers (#43711) 2026-10-12 15:40:06 +08:00
Mohammad Miadh AngkadandMohammad Angkad 06887827c7 [CI] Move tests that do not need GB300 off the per-PR GB300 suite (#43784)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-12 00:38:10 -07:00
5ca0695aa6 [HiCache] Preserve all MTP KV and recurrent state (#40223)
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-12 15:37:59 +08:00
Shuwen Wang f5f5003323 [HiCache] Support Mamba in buffer-only host memory mode (#36345) 2026-10-12 00:35:16 -07:00
Shangming CaiandClaude Fable 5.1 4195980f5a [PD] Poison the deferred abort ack when a Mooncake send fails (#43719)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 15:33:09 +08:00
da4bf1ef8e [Perf] Route BF16 mixed-batch decode tails through decode attention (#43238)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-12 15:27:22 +08:00
Xinyuan Tong a0e6effe50 [Fix] Link cuda_ipc JIT against CUDA driver stubs dir (#43764) 2026-10-12 15:26:32 +08:00
Baizhou Zhang 62c853b654 docs: clarify prefill CP terminology and simplify wording (#43787) 2026-10-12 00:24:57 -07:00
208df292e5 [diffusion][xpu] FLUX.2: enable the bit-exact Triton SwiGLU on XPU (#41940)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 15:16:58 +08:00
Mohammad Miadh AngkadandMohammad Angkad 166b4ae7fc [Fix] Adapt to transformers 5.18 / 5.19 changes (#43688)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 23:24:57 -07:00
ashwini rathi 8ce76b88df [XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast (#42048) 2026-10-12 14:24:26 +08:00
Mohammad Miadh AngkadandMohammad Angkad 130e11d50e [Deps] Bump FlashInfer to 0.7.1 (#43494)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 23:23:11 -07:00
sglang-botandsglang-bot f9bd83e7a9 docs: sync LMSYS SGLang blog cards (#43462)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-12 06:07:15 +00:00
Liangsheng Yin 9e0b001db4 [CI] Skip eval retries in the PD transfer-failure test (#43769) 2026-10-11 22:37:40 -07:00
Liangsheng Yin 2240049194 [RL] Return a failed weight update when the cache flush is refused (#43758) 2026-10-11 21:57:58 -07:00
a297cd3da1 [Fix] scheduler: stop latching batch_is_full when nothing can run (#41872)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-11 21:56:19 -07:00
Mohammad Miadh AngkadandMohammad Angkad bf703013f2 [Test] Re-anchor the VLM flashinfer throughput floor (#43762)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 21:34:50 -07:00
d434ea444d [DSV4.1] Reduce eager Engram extend preparation (#42159)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:31:55 -07:00
Liangsheng Yin a1001bcf3a [PD] Retract transferred decode requests on a retract pause (#43757) 2026-10-11 21:29:39 -07:00
ChengYM 0d0be8fc98 [AMD][MORI] Add MORI EP V2 support (#40666) 2026-10-11 20:54:13 -07:00
Shunkangz 20c49f12c9 Support kv cache sharding for DSA indexer (#40925) 2026-10-12 11:50:23 +08:00
37d2e5c34f [AMD][V4.1][*/N] Fix DP attention + TP MoE decode NaN and support bounded replay (#43077)
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-11 20:47:13 -07:00
weireweire 296774da37 [Tokenizer] Require fastokens>=0.3.2 and include it in the all extra (#43104) 2026-10-11 20:41:53 -07:00
adenzhou1350 cd8ed1bb55 [Bugfix] Preserve available host memory for integrated MUSA devices (#43398)
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
2026-10-11 20:34:58 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 6ac8602e8d [CI] Fix TestQwenImageModulationCache after Qwen-Image modulation sharding (#43747)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:31:29 -07:00
77a15cf4e2 [MemCache] Add shared component size and path APIs to the tree core interface (#43722)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-11 20:14:07 -07:00
95ce9caaf2 [Rust] Lock hyper 1.12.0 and want 0.3.2 in the rust workspace (#43724)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-11 20:12:55 -07:00
e4cfc0c42a [MemCache] Split the SWA mapping move out of locked-full recovery as RecoverEvictedSWA (#43721)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-11 20:11:52 -07:00
WangHemeng 0c52cb1519 [MoE] Skip block-padding rows in the TMA down activation and quant (#42212) 2026-10-12 11:10:30 +08:00
Alison Shao 1a7af3290d Fix Mistral-native format detection when the Hub is unreachable (#43446) 2026-10-11 19:59:19 -07:00
Chunan Zeng 3c74f68a9c [Fix] Initialize BF16 GEMM dispatch in the one-batch benchmark (#43702) 2026-10-12 02:53:02 +00:00
WenhaoZhangandMick Qian d0a098b2e0 [diffusion] perf: split and batch LTX-2.3 one-stage guidance passes under CFG parallel (#43684)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 09:36:01 +08:00
WenhaoZhang a954b395e6 [diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder (#43683) 2026-10-12 09:34:35 +08:00
WenhaoZhang a67228aa43 [diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP (#43691) 2026-10-12 09:32:52 +08:00
Liangsheng Yin e6072935bf [DSpark] Apply SGLANG_SIMULATE_ACC_LEN inside the accept instead of switching paths (#43623) 2026-10-11 15:33:24 -07:00
Liangsheng Yin a65b776963 [DSpark] Add SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot (#43670) 2026-10-11 15:18:59 -07:00
Liangsheng Yin 3cc23b3b86 [DSpark] Merge the draft TP sync sites into dspark-draft (#43669) 2026-10-11 15:17:47 -07:00
Liangsheng Yin 00ba90fd1a [DSpark] Run the same markov block in the draft graph tail; drop SGLANG_DSPARK_FOLDED_SAMPLING (#43668) 2026-10-11 15:15:59 -07:00
Liangsheng Yin 48abbdb845 [DSpark] Draft the eager markov block through MarkovSampleStep (#43622) 2026-10-11 15:13:02 -07:00
Liangsheng Yin 467209e838 [DSpark] Add MarkovSampleStep, one op for a markov draft step's add and token pick (#43621) 2026-10-11 15:10:58 -07:00
Liangsheng Yin 9943cf8eef [DSpark] Run every markov head through one step interface (#43620) 2026-10-11 15:07:51 -07:00
Banruo LiuandShangming Cai d62b9c2480 [KV Events] Publish LoRA adapter names and namespaced block hashes (#42455)
Signed-off-by: Banruo Liu <lichangxinlivingshade@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-11 23:23:30 +08:00
97bfa5cc53 [diffusion] fix: select VSA-H3 for FastH3 when no transformer backend is given (#43690)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:05:25 +08:00
Ke Bao 46a97e5540 Preserve serialized page dtype in NIXL copy fallback (#43593) 2026-10-11 22:39:14 +08:00
Ke Bao b0a75c8196 Share MLA host capacity initialization (#43595) 2026-10-11 22:36:29 +08:00
Ke Bao c938b6e454 Build Mamba host pools from device buffer infos (#43596) 2026-10-11 22:35:52 +08:00
Shuwen Wang ef3e5c83c5 [HiCache] Keep storage workers alive on backend exceptions (#43550) 2026-10-11 22:26:56 +08:00
MickandMick Qian 11bfd868ac [diffusion] perf: encode LTX video/audio through the shared bounded MP4 path (#43656)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 21:15:49 +08:00
Shuwen Wang 0923bcd258 [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking (#43567) 2026-10-11 21:12:51 +08:00
FeiHong Guo cf88ed4352 [BugFix][DP] Prevent health checks from perturbing user request routing (#35721)
Signed-off-by: Phoenix3334 <Phoenix3334@users.noreply.github.com>
2026-10-11 21:08:55 +08:00
Shuwen Wang d26b6a5a9c fix: default Inkling to fa4 on Hopper (#35292) 2026-10-11 20:57:42 +08:00
trebladevandMick Qian 586d98cc15 [Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5 (#43062)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 20:55:59 +08:00
MickandMick Qian 04f740580c [VLM] fix: repair multimodal VMM lease cleanup and source-stream ordering (#43475)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 19:13:39 +08:00
efd14cfb66 [DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel (#43070)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yuan Luo <4908075+yuan-luo@users.noreply.github.com>
2026-10-11 19:03:19 +08:00
Sage 53295c6972 [kvevents] fix missing topics in replayed kv events (#43386)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-11 17:25:43 +08:00
Liangsheng Yin 0a21a3b798 [Scheduler] Keep a DP-converted decode batch as the running batch (#43283) 2026-10-11 01:23:51 -07:00
jiaqiang-dot-liu 41652225d0 [Sampler] Sort only the max(top_k) leading columns in the torch sampler (#43346) 2026-10-11 01:00:59 -07:00
Liangsheng Yin dab108b9f3 [Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool (#43290) 2026-10-11 00:18:27 -07:00
Liangsheng Yin 89e24e7a20 [KV canary] Expect sweep counters on the target runner only (#43286) 2026-10-10 23:52:17 -07:00
Liangsheng Yin ad8b1e1f85 [Test] Do not hang scripted-runtime teardown when the server died (#43285) 2026-10-10 23:13:56 -07:00
Liangsheng Yin 448759b235 [Scheduler] Requeue preempted requests when admission admits nothing (#43298) 2026-10-10 22:46:55 -07:00
Qiaolin Yu 047405f534 [qwen 3.8 next] Fuse QSA prefill KV gathering and dtype conversion (#41933) 2026-10-10 22:40:23 -07:00
paulzhang-tm 5678b71b6c [Spec] Allow Qwen3.5 backends to opt into packed target verification (#43037) 2026-10-10 22:39:17 -07:00
Nikolai 22d63f5e43 [AMD]Qwen add Sparse attention to capturing hip graphs (#43383) 2026-10-10 22:03:54 -07:00
Jzz1943 260551ebe9 [KDA] Support Helion MTP verification and harden prefill, decode (#42857) 2026-10-11 12:47:57 +08:00
SunskyXHandXiaoyu Zhang 65e6bb6be7 [GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout (#43332)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 12:43:16 +08:00
a1500b10e3 [diffusion] Don't report video jobs completed after their outputs are deleted (#43084)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 12:42:38 +08:00
Chunan ZengandClaude Opus 5.5 aa5c969dc3 [Perf] Fuse BF16 writes to native HND KV cache storage (#43237)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:41:18 +08:00
Shuwen Wang 58cb1a2be6 [Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks (#43479) 2026-10-10 21:18:41 -07:00
Avaya Aggarwalandniehen6174 3458c94fad [diffusion] comfyui: consolidate executor conditioning-cache fixes (#43492)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-11 11:42:00 +08:00
Avaya Aggarwalandniehen6174 638330d975 [diffusion] comfyui: consolidate FluxAdapter correctness fixes (#43490)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-11 11:41:36 +08:00
Khoa Pham 6576e50cd5 [1/N][DSv41] Enable DP attention with MegaMoE (#43228) 2026-10-10 20:36:35 -07:00
b931b2c8bd [diffusion] perf: run the attention section eager under torch.compile for MiniMax-H3 (#43634)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:02:54 +08:00
b21826e609 [diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths (#43633)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:55:01 +08:00
b237afa547 [diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad (#43632)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:54:18 +08:00
b3a65efe7e [diffusion] perf: keep Cosmos3's fused kernels in the graph (#43577)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:53:43 +08:00
d45ef0f2d2 [diffusion] perf: keep Qwen-Image-2.1 and Ulysses graphs whole (#43586)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:52:43 +08:00
Shangming CaiandKevin Flansburg e1f43c9a61 [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll (#43562)
Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com>
2026-10-11 10:12:34 +08:00
Richard Wang d60e0c8570 [diffusion] Support LLaDA-Image BF16 generation and editing (#37907) 2026-10-11 02:10:51 +00:00
e4b423256e [diffusion] fix: serve whole graphs after offload-during-compile (#43631)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:03:47 +08:00
Richard Wang d7ed5b56f5 [diffusion] fix: keep the overlay probe from stalling cached startup (#43590) 2026-10-11 10:01:01 +08:00
156d54e56c fix: wait for HiCache load-back before hybrid MLA KV reads (#38217)
Co-authored-by: zhouzechun <zhouzechun@kuaishou.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-10-11 09:50:13 +08:00
Jzz1943andXiaoyu Zhang 9336681310 [DSA] Support TileLang FP8 KV cache on CUDA (#42957)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 09:12:18 +08:00
Xiaoyu Zhang 33d2ef1756 [Fix] Reserve remaining tokens for ignore_eos requests during prefill admission (#43564) 2026-10-11 09:05:48 +08:00
WenhaoZhangandOnePunchMonk 5730345717 [diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode (#43495)
Co-authored-by: OnePunchMonk <aggarwal.avaya@yahoo.com>
2026-10-11 08:58:23 +08:00
d1313dd5fe [diffusion] fix: skip inductor's saved multi-kernel pick for torch.compile (Z-Image warm starts crash or hang) (#43601)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:56:55 +08:00
3022989bad [diffusion] CI: skip the LTX-2 RMSNorm-modulation guard test off the CUDA platform (#43574)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:54:37 +08:00
65386cb95a [diffusion] perf: add the dynamic LoRA delta inside the second GEMM (#43501)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:50:53 +08:00
e328c7ecbd [diffusion] fix: skip QKV packing while layers are offloaded (#43576)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:47:35 +08:00
7373665085 [diffusion] fix: keep exported buffers out of expandable segments for IPC all-to-all(#43584)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:46:30 +08:00
5e872557cb [diffusion] feat: keep the TP all-reduce inside the graph (#43602)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:45:54 +08:00
664e4c6499 [diffusion] feat: warn instead of disabling on unvalidated models for BCG(#43568)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:40:28 +08:00
23764b1242 [diffusion] feat: bind each GPU worker to its GPU's NUMA node (#43565)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:38:58 +08:00
9933cf5437 [diffusion] feat: declare regional compile regions for flux (#43575)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:33:29 +08:00
68e1a3dbb1 [diffusion] fix: accept request-level profiling on the image API (#43573)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:30:20 +08:00
d32d8fa0d1 [Dsv4.1] Mega-mhc integration (#42808)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-10 17:25:17 -07:00
Liangsheng Yin 7187bb593e [Model] Keep gpt-oss router logits batch-invariant under deterministic inference (#43624) 2026-10-10 17:22:29 -07:00
Mohammad Miadh AngkadandMohammad Angkad c640dc97ce [CI] Fix Kimi-Linear DCP PD test startup (#43598)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-10 16:53:54 -07:00
Cheng Wan 234ccba7f6 [Refactor] Declare every model's stages from a shared function and drop the meta-device neighbour build (#43460) 2026-10-10 12:43:09 -07:00
Cheng Wan 3b16eeedce [Refactor] Declare pipeline neighbours from a shared declaration function (#43459) 2026-10-10 12:42:46 -07:00
Cheng Wan 3b717c0f13 [Refactor] Decide an exit's fixed facts when the stage is built (#43458) 2026-10-10 12:42:23 -07:00
Cheng Wan a956aa192a [Refactor] Bind a layer stack in one pass over each stage's data flow (#43457) 2026-10-10 12:42:00 -07:00
Cheng Wan d2c597c8db [Refactor] Decide input-scattered eligibility and stage rejections in one place (#43456) 2026-10-10 12:41:38 -07:00
Cheng Wan b0dde87de2 [Refactor] Make residual read and update capabilities explicit and checked (#43455) 2026-10-10 12:41:16 -07:00
Cheng Wan 514feb1afc [Fix] Keep residual rows and sums consistent across appends and pipeline handoffs (#43454) 2026-10-10 12:40:53 -07:00
Cheng Wan 8bd7f2d178 [Refactor] Record what a bound stage path does instead of reading it back (#43453) 2026-10-10 12:40:30 -07:00
0acd93c35c [Scheduler] Use committed length in spec mamba track boundary check (#43029)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-10 12:29:40 -07:00
Shuwen WangandKe Bao f9cee8d2b2 [HiCache] Keep protected cgroup page cache charged in host budgets (#42788)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-10-10 17:03:40 +00:00
Xiaoyu Zhang f61163b4b0 [Docs] Let the Qwen3 B300 recipe use the default attention backend (#43554) 2026-10-10 10:00:04 -07:00
Xiaoyu Zhang 587b6d622d [Docs] Document Qwen3-4B/8B H200 mixed-chunk latency tradeoffs (#43551) 2026-10-10 09:59:46 -07:00
Shunkangz de487f8039 [kv-shard 3/4] Enable Control Plane C (#40102) 2026-10-10 22:53:58 +08:00
huangtingwei 406540ab5f [HiCache] Add staged write-back for the page-unified KV cache layout (#39606) 2026-10-10 22:49:45 +08:00
Shuwen Wang 99d6f4e158 [MemCache] refactor: require an anchored lock receipt for non-root releases (#41452) 2026-10-10 22:43:49 +08:00
Shuwen Wang 44bc478863 [HiSparse] Avoid host synchronization in eager backup (#41446) 2026-10-10 21:49:55 +08:00
hhy-seven 616178a596 [Responses] Fix max_output_tokens overestimate on multimodal requests (#35524) 2026-10-10 13:42:53 +00:00
a510340aeb [diffusion] perf: run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout (#43508)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:34:54 +08:00
72657f7ca5 [diffusion] perf: pipeline every USPAttention Ulysses path's exchange over the copy engine (#43521)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:34:18 +08:00
Shuwen Wang 416ebe6090 [CI] Fix EAGLE3 CUDA parity prefill chunk alignment (#43538) 2026-10-10 21:11:17 +08:00
49e629d239 [Rust] Upgrade sglang-radix-tree to PyO3 0.29 (#42807)
Co-authored-by: colleen-tml <296620403+colleen-tml@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:56:50 +08:00
Bingxu Chen a916d91cd3 [AMD][CI] Cap pyarrow below 26 for lmms-eval (#43547) 2026-10-10 18:59:49 +08:00
Sage 6fc8d9da32 [rust-server] preserve PD routing in chat and completions (#42989)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-10 17:29:46 +08:00
Liangsheng Yin c55790d38b [CI] Drop CPU tests with no bug-catching signal and run Rust workspace tests in their own workflow (#43503) 2026-10-10 02:13:53 -07:00
siliangchen-amd ff19159e16 [AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears (#41071)
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com
2026-10-10 01:31:20 -07:00
chuyeh e35f60b9bc [AMD][CI] Drop removed tp_rank/tp_size kwargs from small-M FP8 proj test (#43318) 2026-10-10 00:52:28 -07:00
d7d22e8a30 [AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size (#39840)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-10 00:47:13 -07:00
d8c5eca3ac [diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away (#43385)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:02:03 +08:00
Richard Wang 441de659f4 [CI] Fix Python 3.10 failure in prefill complete integration test (#43505) 2026-10-10 00:00:03 -07:00
tao 6e8f3e09dd [Model Loader] Add InstantTensor loader support (#40453) 2026-10-09 23:01:59 -07:00
cctry 33f6b00f32 Clamp default decode CUDA graph max_bs to max_running_requests (#43172) 2026-10-09 22:54:32 -07:00
Mohammad Miadh AngkadandMohammad Angkad 16be9df25f [CI] Remove the GLM-5.3-Flash PTX KDA prefill nightly test (#43470)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 22:52:17 -07:00
Yuwei An 8b2aaf7060 [Refactor][TCPCG] Retire backend selection and configuration (3/9) (#42287) 2026-10-09 22:42:35 -07:00
Yuwei An 0aa580f904 [Dsv4.1] Add DeepSeek V4.1 DSPARK TP and DP EP accuracy coverage (#43478) 2026-10-09 22:41:30 -07:00
YAMY 17852cdfe8 [DSv4.1] Initialize the full-to-SWA mapping for request-window pools (#43165) 2026-10-09 22:39:26 -07:00
Frederik Gossen 3831e7e091 [diffusion] Z-Image: autotune the shared transformer block at both token scales (#35755) 2026-10-10 13:04:43 +08:00
Raymond HuangandClaude Opus 5 aa79087017 [diffusion] fix: fail MiniMax-H3 quality="high" closed on substituted DiT weights (#34978)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-10 13:03:44 +08:00
ccae3b3639 [diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell (#43373)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:58:30 +08:00
Shangming Cai 1814ece57f [PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages (#43329) 2026-10-10 12:38:38 +08:00
cctry 20948cdcc6 [PD] Reclaim parked optimistic prefills when request slots are exhausted (#43442) 2026-10-09 20:31:24 -07:00
Liangsheng Yin 7d44269907 [MemCache] Fix the strict busy check on prefill-aware SWA and tidy small leftovers (#43474) 2026-10-09 20:27:27 -07:00
f2e9a2bed1 [unified memory pool] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts (#43206)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-09 20:19:30 -07:00
MickandMick Qian a65f4ffba5 [diffusion] docs: standardize diffusion cookbook command builders (#43472)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 11:14:31 +08:00
436d73ddda [diffusion] model: support HunyuanImage-3.0-Instruct (#36311)
Co-authored-by: Elizaveta Martirosian <you@example.com>
Co-authored-by: wes-lyu <294033513+wes-lyu@users.noreply.github.com>
Co-authored-by: Wes <its.wesley.lyu@gmail.com>
Co-authored-by: AuFlow <73925903+AuFlow@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-10 11:11:28 +08:00
506f140166 [sgl-router] Read finished /generate streams to the end; expose reorg admission caps as flags (#43233)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 10:59:09 +08:00
Benjamin TruongandXiaoyu Zhang 37678784c6 [Gemma4][Attention] Enable Gemma4 FA4 on SM90 with head-dim 512 support (#42019)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-10 10:40:19 +08:00
55a8dffc87 Skip the eager activation reserve in KV sizing on a capped Full prefill role (#43435)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-09 19:36:48 -07:00
a338b9858a [PD] Receive decode KV into separate paged SWA host pools; let host-capable receivers own abort drain tracking (#43410)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 19:33:06 -07:00
Vladislav NosivskoyandBBuf 2ff3e6dce1 [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 (#42604)
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-10 10:29:42 +08:00
Liangsheng Yin 465253ef52 [Scheduler] Fix the deterministic-inference hang when a prefill chunk is shorter than its alignment (#43444) 2026-10-09 19:06:18 -07:00
78218194ad perf(kda): rebalance MTP verify precompute and state synchronization (#42683)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-10 10:03:19 +08:00
Michael 37114afe20 [AMD][Fix] Saturate FP8 KV commits instead of reproducing e4m3fnuz NaN overflow (#42982) 2026-10-10 09:49:56 +08:00
MickandMick Qian 77ba54f7df [diffusion] docs: add Wan-Animate-2 cookbook command picker (#43380)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:45:16 +08:00
16423cb6d2 [sgl-router] Serve DeepSeek-V4.1 Flash through sglang-processor (#43000)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:23:09 +08:00
5f94a45896 [srt] Allow mixed chunk prefill on the Full prefill CUDA graph backend (#43461)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-09 18:21:22 -07:00
MickandMick Qian cde16aa64d [diffusion] model: support Qwen-Image 2.1 Turbo and checkpoint sampling grids (#43391)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:12:56 +08:00
e8fb52c526 [Fix] Keep request-pool row generations monotonic across clear() (#43439)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: colinchan15 <colinchan15@users.noreply.github.com>
2026-10-09 18:08:54 -07:00
ashwini rathi 8cf4d73c84 [XPU] Fix XPU CI breaks from #41105 (KDA chunk_offsets) and #42588 (int4 test tp_group) (#43334) 2026-10-10 09:06:24 +08:00
Avaya Aggarwal 66e6b68d82 [diffusion] comfyui: report the real runtime import error (#43178) 2026-10-10 09:05:12 +08:00
ff3aa58f11 [diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per war while bit-identical (#43337)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:03:36 +08:00
bookinstockandMick Qian 32f34e898b [diffusion] fix: honor requested inference step count for MiniMax H3(#38671)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:02:57 +08:00
bd7c7971ba [diffusion] perf: move MiniMax-H3's pipelined Ulysses copies off the SMs (#43361)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:58:07 +08:00
Alison Shao 9b37e28b83 Disable DualGemm fusion for Inkling dense MLP (#43447) 2026-10-09 17:56:20 -07:00
6acaf3374a [kernel] SwiGLU-OAI MXFP8 producer; move swizzled MXFP8 producers to ops/quantization (#42810)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Denji-kk <51020465+Denji-kk@users.noreply.github.com>
2026-10-10 08:51:57 +08:00
Rain Jiang 961bcf481d support DeepEP v2 on GLM-5.3 (#43432) 2026-10-09 16:57:12 -07:00
Yuwei AnandYuwei An 2d38934493 [DSv4.1] mxfp8 dispatch cache (#42156)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
2026-10-09 16:36:28 -07:00
Liangsheng Yin 46498bc8c5 [MLX] Sync decode KV to the pool once at release, clamped to the owned prefix (#43292) 2026-10-09 16:20:45 -07:00
Liangsheng Yin 114cb44920 [MLX] Read radix prefix slots from the model runner's req_to_token pool (#43291) 2026-10-09 16:18:58 -07:00
Liangsheng Yin 8020ea2ee9 [MLX] Drop a retracted request's runner state at its re-prefill (#43284) 2026-10-09 16:16:59 -07:00
Liangsheng Yin f7e601b936 [MLX] Drain in-flight jobs before a retract-mode pause_generation (#43282) 2026-10-09 16:14:23 -07:00
f6c848c882 [PD] Use 1 to disable decode host receive and 0 for always-on (#43407)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 15:50:52 -07:00
Alison Shao f6e3444b2e ci: diffusion-only option for the runner utilization report (#41620) 2026-10-09 15:47:47 -07:00
Cheng Wan 58f6976ce7 [Fix] Unblock CUDA graph startup for hybrid parallelism (#43211) 2026-10-09 15:39:59 -07:00
cb9bd1d4d4 [Perf] Reduce multi-detokenizer router IPC sends (#42238)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-09 15:32:14 -07:00
66d840a8ef [cuda_graph] Size dp-local decode graphs by the runner's own gather requirement (#43303)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-09 15:21:23 -07:00
Kevin MiandClaude Opus 5.5 6d8ddaf1d7 [AMD] MiniMax-M3 indexer CP: packed scoring for EAGLE verify rows (#42614)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:05:50 -07:00
fbf8d730fd [Fix] MiniMax-M3: default to the breakable prefill CUDA graph (#41845)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:05:05 -07:00
Kevin MiandClaude Opus 5.5 70e82f7b73 [AMD] Run the MiniMax indexer CP test on MI35X, not MI300 (#43259)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:04:25 -07:00
weireweireandweireweire 46c2585faa [DSpark] Fold draft sampling for DeepSeek-V4.1 heads under AUTO (#42987)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-10-09 14:40:28 -07:00
Huabin 2f1acef3d8 [Model] Add LLaDA2.2 Block Routing MoE support (#31768) 2026-10-09 12:14:44 -07:00
Mohammad Miadh AngkadandMohammad Angkad 72eab59bf6 [GLM-5.3-Flash] Enable breakable prefill CUDA graph by default (#42845)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-10 02:41:02 +08:00
87845cda2b [Multimodal] Chunk and cache audio items whose offsets span several placeholder runs (#43301)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com>
2026-10-09 10:57:01 -07:00
3bffe69b74 [moe] Fill num_token_non_padded for the customized A2A backend at EP 1 (#43245)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 10:55:02 -07:00
3c612132fd Overlap scheduler startup with the data parallel controller (#43177)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
2026-10-09 10:54:22 -07:00
Mohammad Miadh AngkadandMohammad Angkad 5cbf949839 Fix torch.compile crash in fused gate-sigmoid-mul launcher (#43384)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 23:51:30 +08:00
76ccf0850c [diffusion] fix: fix TeaCache CFG state lifecycle and skip boundaries (#42422)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-09 23:20:13 +08:00
121aa8dcc7 [diffusion] perf: fuse MiniMax-H3's RMSNorm and indexed AdaLN under quality lossless (#43327)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:39:11 +08:00
Jimmy Shong 408f5d057a docs: add Clef accuracy results to the deployment wizard (#43349) 2026-10-09 22:19:27 +08:00
1203eb9181 [diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against attention over the copy engine (#43164)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:04:29 +08:00
xiaofei-zheng fba3ad4b1f [Frontend] Parallelize long chat prompt encoding (#41259) 2026-10-09 21:30:22 +08:00
xiaofei-zheng db32673b78 [AMD] Let the GLM DSA NextN draft declare its own shared-expert fusion architecture (#41258) 2026-10-09 21:28:45 +08:00
xiaofei-zheng d5c3ae9752 [AMD] Eliminate remaining gfx95 FP8 scale relayout copies (#41030) 2026-10-09 21:27:38 +08:00
78996e3f02 [Performance] Batch Mooncake hybrid-pool metadata and puts (#39696)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:12:42 +08:00
d511e6bd1d [sgl-router] Parse tool calls when serving chat through /generate (#42999)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:07:30 +08:00
67d063f63b [diffusion] feat: allow per-role attention backend override (#35310)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 20:57:29 +08:00
348689da40 [PP+PD] Fix #38206 Reduce bootstrap consensus latency (#38959)
Co-authored-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 20:53:48 +08:00
f1e0472fcc [diffusion] model: support Wan-Animate-2 (#42941)
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 20:46:23 +08:00
Shuwen Wang 0955169ed2 [mem_cache] Bound FlexKV checkpoint stores and restore DSPARK cache checks (#35694) 2026-10-09 20:25:33 +08:00
59eb71a831 [diffusion] fix: align SGLD CFG combine formula with diffusers Z-Image (#23772)
Co-authored-by: SGLang CI <ci@sglang.ai>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-09 19:47:03 +08:00
Shuwen Wang f48ed2127e test: allow bounded checkpoint lag in Mamba HiCache tests (#42523) 2026-10-09 19:24:39 +08:00
3805d78796 [sgl-router] Serve /v1/chat/completions through /generate, with reasoning parsing (#42998)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-09 19:17:52 +08:00
Xiaoyu Zhang 0eb24e7d5b [Performance] Optimize Qwen3.8-Flash-Next BF16 decode on Blackwell (#42175) 2026-10-09 18:46:12 +08:00
516fd1c1b5 [diffusion] perf: tile MiniMax-H3's indexed modulation kernels by 2048 columns (#43267)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 18:23:52 +08:00
Xinyuan Tong 91b3692a4c docs: add thinking budget guide (#41601) 2026-10-09 17:50:34 +08:00
BingjiaWang 888de24cc8 Add BJWang-ant to CI_Permission (#43321) 2026-10-09 02:31:38 -07:00
黄孝君 e1c6ddca1d [NPU] Bump SGLANG_KERNEL_NPU_TAG to 2026.9.0.post9 (#43340) 2026-10-09 17:30:38 +08:00
vOvandXiaoyu Zhang 7cfecbef56 [diffusion][model] Add native SANA-Video 2.0 support (#41492)
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-09 17:28:03 +08:00
DevashishLal-CB 4debae7534 Add blackwell dual gemm fusion support for MLP (#41887)
Signed-off-by: Devashish Lal <devcode@meta.com>
2026-10-09 17:26:33 +08:00
Thomas Wang 5eca4d8caf [AMD] Set maxseq length limit for DSV4.1 BCG metadata buffers (#42570) 2026-10-09 02:15:43 -07:00
Xinyi Song 23f890613e [AMD] Patch aiter's group32 MXFP8 GEMM masked-scale fix (#43319) 2026-10-09 02:08:44 -07:00
Jimmy Shong 14e3414513 [Docs] Add Clef cookbook with selectable Clef Flash variant (#43235) 2026-10-09 01:56:45 -07:00
amote-i ff146fa20c [NPU] [DOC] remove --enable-torch-compile from Ascend support matrix (#43336) 2026-10-09 16:55:39 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7578432c21 [LoRA][Test] Use the 0.9 ROUGE-L tolerance for the CUDA multi-LoRA batch test (#43297)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 01:51:01 -07:00
Avaya AggarwalandMick Qian d677848968 [diffusion] fix: fix CFG normalization to use per-sample norm instead of batch-flattened norm (#41706)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:43:18 +08:00
Murthy LandClaude Opus 5 3ab142adc4 [diffusion] feat: enable breakable CUDA graphs for FLUX.2 Klein (#36470)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-09 16:40:09 +08:00
b922f5ca20 [diffusion] fix: share the IPC all-to-all transport between AllToAll4D and USP (#43269)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:31:55 +08:00
MickandMick Qian 17b2f35ae7 [diffusion] refactor: remove unused helpers and write-only state (#43064)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:30:55 +08:00
James XuandMick Qian ee4e2f7818 [diffusion] optimization: enable channels_last_3d for Cosmos3 VAE (#28021)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:29:46 +08:00
Cheng Wan dc0b394471 [Refactor] Remove Kimi K3's own layer communication and let the consumer set an FFN's exit (#42936) 2026-10-09 01:27:46 -07:00
Cheng Wan 305ad5503d [Refactor] Run Kimi K3's fused collectives through stage boundaries (#42898) 2026-10-09 01:27:23 -07:00
Cheng Wan 2b469040d2 [Refactor] Support attention-TP dense FFNs and unpadded batches on stage boundaries (#42897) 2026-10-09 01:27:00 -07:00
Cheng Wan b9ae3d1a65 [Fix] Run attention on a padded extend batch's real rows (#42896) 2026-10-09 01:26:38 -07:00
Cheng Wan 7203747cc8 [Fix] Shard Kimi K3's shared experts over the TP group they sum over (#42895) 2026-10-09 01:26:15 -07:00
MickandMick Qian 45ad463c87 [diffusion] refactor: deduplicate UniPC solvers and Gemma3 shard loading (#42899)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:26:04 +08:00
Cheng Wan 125f88cb75 [Refactor] Run Kimi K3 through stage boundaries (#42894) 2026-10-09 01:25:51 -07:00
Cheng Wan 666da4f56b [Fix] Kimi K3 KDA value head size and SiTU input alignment (#42893) 2026-10-09 01:25:28 -07:00
Cheng Wan 48a6c0a5d0 [Fix] Return an FFN's output to the attention rows at a pipeline handoff (#42892) 2026-10-09 01:25:05 -07:00
Cheng Wan b2402ccad6 [Fix] Merge DeepSeek-V4 two-batch-overlap microbatches without a residual (#42891) 2026-10-09 01:24:42 -07:00
b130f8af02 [sgl-router] Serve /v1/completions through /generate, with the shared OpenAI layer (#42997)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:04:04 +08:00
f620d733d2 [rust-processor] Share the reasoning splitter and tool constraint across hosts (#42996)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-09 15:19:41 +08:00
37ae292e6f [diffusion] fix: skip LTX-2 SP masks only for globally unpadded sequences (#43134)
Co-authored-by: yatholam <yatholam@amd.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 15:08:41 +08:00
Lianmin Zheng c44a42e6d3 docs: reorder README workloads, hardware, and ecosystem (#43307) 2026-10-08 23:46:24 -07:00
e0608d890e fix: use dp_size instead of attn_dp_size for msprobe rank id (#43041)
Co-authored-by: l30044004 <louyujing@huawei.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-09 09:40:43 +03:00
Gabriel Wu 217980566a fix(fa4): isolate sheared-bias workspaces by CUDA stream (#33000) 2026-10-08 23:37:02 -07:00
areneau-cfandJimmy Shong bd2d73daa5 [model] support Clef and Clef-Flash decision models on /v1/systemone (#42721)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-09 14:25:52 +08:00
Michael 51f7ea84c7 [AMD] Keep MiniMax decode-score tile at 128 tokens off gfx950 (#42772) 2026-10-08 23:22:19 -07:00
08fc290f48 [AMD] gfx950 small-batch MoE: expert-count gate for the small sort and FP8 block-scale small-M kernel (#41982)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 23:05:50 -07:00
cba1d07a94 [Memory] Share one full prefill graph output buffer across shapes for tuple and list outputs (#43176)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-08 23:05:29 -07:00
e14ef4b658 [Scheduler] Skip the decode for a request whose last token is already in flight (#42720)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-08 23:00:46 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 1fc6b8962c [Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 (#43083)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:55:35 +08:00
e46aaaa789 [diffusion] feat: accept strided row views in the packed QKV all-to-all (#40588)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 13:55:31 +08:00
Colin Z 3c1703d2ff [AMD] Support Quark DeepSeek-V4-MXFP4 checkpoints on the DSV4 stack, enabling shared-expert fusion and MTP (#31054) 2026-10-08 22:54:49 -07:00
Yuan Luoandluoyuan.luo 9a1aeacc3b [GLM-5.3-Flash] Capture the KDA extend in breakable prefill CUDA graphs (#41105)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-10-09 13:37:20 +08:00
33e7b4cc4b [diffusion] perf: decode plain base64 MiniMax-H3 materials without a per-character loop (#43117)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:25:23 +08:00
Chetan Kumar VermaandClaude Fable 5.1 5d8e98b7fd [XPU][Diffusion] Plain-torch scale-shift for inductor (#41885)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-09 13:07:14 +08:00
saatwiknagpal 6e7beace14 [diffusion] fix: fix int32 offset overflow in the Ulysses packed-QKV pack kernel (#43140) 2026-10-09 11:51:46 +08:00
3ba90f1b52 [rust] Update Dynamo frontend crates to latest compatible releases (#42666)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:46:01 +08:00
9172a73725 [sgl-router] Keep charset=utf-8 on streamed responses (#43232)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:44:52 +08:00
Baizhou Zhang 33bffcd490 [CP] GLM-5.3-Flash Prefill CP implemented on layer boundaries (#42044) 2026-10-08 20:42:11 -07:00
metamergebotandjoshuadeng 7859eff6ca [srt] Inject "__req__" into custom_params only for custom logit processors (#43181)
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-08 20:40:50 -07:00
4ee3f0f370 [Session] Skip the KV free when page alignment empties a streaming-session rewind (#43183)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Joshua Deng <91448271+joshuadeng@users.noreply.github.com>
2026-10-08 20:40:25 -07:00
4e3eaba619 [HiCache] Re-admit a split node's queued buffer-mode write instead of dropping it (#43174)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 20:39:41 -07:00
HEJIAN SANGandClaude Opus 5.5 3e13f1b2c7 [Perf] Serialize OpenAI chat/completion responses in one pass (#42889)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:17:25 +08:00
ormandjandkpham-sgl 438df9a2a4 [Spec] Fix wrong tokens from compact-verify CUDA graphs when a token tier is captured more than once (#42709)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-08 19:49:23 -07:00
Khoa Pham 8e19db19b0 [KDA] Default safe-gate Blackwell prefill to FlashInfer (#43246) 2026-10-08 19:46:28 -07:00
Chetan Kumar VermaandClaude Opus 5.5 fedab5537b [XPU] Enable ring attention on XPUAttentionBackend (#41406)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 10:41:11 +08:00
MickandMick Qian df1e4f3f2b [diffusion] refactor: share SANA-WM refiner block computation (#42980)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 10:38:36 +08:00
Liangsheng Yin f6569bce4d [MemCache] Skip KV events and eviction config on a disabled radix cache (#43253) 2026-10-08 19:27:39 -07:00
Polisetty V R K Jyothendra Varma e54f08c860 [XPU] Misc changes to support XPU (#39817)
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com>
2026-10-09 10:26:46 +08:00
ashwini rathi 51686cf8e7 [XPU] split Intel nightly into kernel-main and kernel-wheel jobs (#41857) 2026-10-09 09:58:39 +08:00
shodoco 86624f14eb [Rust] Pass the media item fps hint to multimodal processors (#43143) 2026-10-08 17:51:06 -07:00
Liangsheng Yin e0ab91e3ca [MemCache] Rename SWAChunkCapPoolConfigurator to SWARequestCapPoolConfigurator (#43202) 2026-10-08 17:50:51 -07:00
b2786a670e [MemCache] Count reclaimable tokens in the unified tree ledgers (#43185)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 17:48:11 -07:00
Liangsheng Yin b4fe282836 [MemCache] Delete ChunkCache (#43201) 2026-10-08 17:31:16 -07:00
Liangsheng Yin 508ba79688 [MemCache] Serve disabled hybrid-SWA models with UnifiedRadixCache (#43200) 2026-10-08 17:30:01 -07:00
Liangsheng Yin 265efe5e01 [MemCache] Serve disabled full-attention models with UnifiedRadixCache (#43199) 2026-10-08 17:27:24 -07:00
Liangsheng Yin 977e2719c7 [MemCache] Serve disabled pure-SWA models with PureSWARadixCache (#43198) 2026-10-08 17:25:54 -07:00
Liangsheng Yin 57ce47862d [MemCache] Report no prefix sharing from a disabled RadixCache (#43197) 2026-10-08 17:23:11 -07:00
Khoa Pham fc3385d530 feat(kda): integrate FlashInfer prefill checkpoints (#41400) 2026-10-08 17:16:33 -07:00
Khoa Pham 1e77db3215 [Perf] Avoid gate copies in gated RMSNorm (#43227) 2026-10-08 17:11:58 -07:00
20d1f3f56a [Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction (#41593)
Signed-off-by: Pei Li <peili@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-08 17:05:04 -07:00
Ankur Singh 2b94cb4eb5 docs(cookbook): add Qwen3.8 GB300 long-context recipe (#43159)
Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
2026-10-08 16:56:30 -07:00
bb43927404 [PD] Charge decode admission the page-rounded allocation when the radix cache is disabled (#43173)
Co-authored-by: hanming-lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 16:48:31 -07:00
Liangsheng Yin 2e64740442 [CI] Avoid Hugging Face Hub API calls when the local model cache is complete (#43229) 2026-10-08 16:27:31 -07:00
Liangsheng Yin baee5f3b62 [CI] Drop the labeled trigger from the extra CI workflows (#43234) 2026-10-08 16:25:50 -07:00
Liangsheng Yin ae4da8cd78 [MemCache] Skip SWA request-cap pool sizing when chunked prefill is off (#43203) 2026-10-08 16:19:35 -07:00
Liangsheng Yin 71584b62c0 [CI] Fix stale fakes in the NVFP4 MoE dispatch and CP strategy unit tests (#43226) 2026-10-08 15:51:39 -07:00
20bf93b6bf [metrics] Call DPBalanceStats.create by keyword in the ratio ladder test (#43218)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 15:24:38 -07:00
2abb761639 [metrics] Count per-rank DP attention pairs and their imbalance (#43180)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 15:01:37 -07:00
metamergebotandHanming Lu 0b658c12ff [Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks (#43188)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
2026-10-08 15:01:16 -07:00
Siyuan Chen b0e06359ac [DSV4.1][PD][5/N] Decode node support DP attention (#40177) 2026-10-08 14:42:16 -07:00
Xuanteng Huang d20bdc1443 Add per-token NVFP4 MoE support for ReLU2 activation (#39199)
Signed-off-by: Xuanteng Huang <xuanteng.huang@outlook.com>
2026-10-08 14:28:22 -07:00
ShunkangzandClaude Opus 5 3d8d44a4a9 [Qwen3.8 CP 3/4] Collocated prefill CP integration (#39723)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-08 14:20:27 -07:00
Sage 5a5ea81036 [rust-server] warm up each rust listener in-process (#43004)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-08 14:09:46 -07:00
e902207a4e [Speculative] Support block verification for the DFlash family (#43012)
Co-authored-by: maocheng <mao.cheng@MacBook-Pro-MD663G60VV.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:59:52 -07:00
d1ba943016 [Model] Add support for EmbeddingGemma 2 (#42782)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:54:45 -07:00
Baizhou Zhang b193ef382a [CI] Clean some unnecessary DSV4 tests (#43195) 2026-10-08 13:53:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad d8448d2e62 [CI] Re-enable GB300 (#43058)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-08 13:23:26 -07:00
metamergebotandhanming-lu 93f2acf614 [Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio (#43187)
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 12:58:58 -07:00
Ming Yang 3fef129ff8 [PD] Allow state-only KV checksum inputs (#42904) 2026-10-08 12:54:03 -07:00
075fd8fa0b [Mamba] Describe the track snapshot dtype by its kernel instead of fp32 (#43192)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-08 12:47:46 -07:00
e3c5719713 [DFlash] Build the attention QKV projection through an overridable hook (#43179)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-08 12:46:28 -07:00
kunchengit 00b3a48753 [dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding (#31773) 2026-10-08 12:44:14 -07:00
Aurick QiaoandQiaolin-Yu b2cb24995d [Debug] Add opt-in NaN detection before sampling (#42690)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:20:57 -07:00
Shenxiu LiuandQiaolin-Yu 393eaed509 [Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather (#34076)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:19:35 -07:00
shodoco 1638c5d699 [http_server] Let custom /generate routes skip the second contract check (#43114) 2026-10-08 11:56:06 -07:00
214347891a [Bugfix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path (#41072)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 18:01:27 +00:00
cbf49695c3 [CI] Add dedicated sglang-processor workflow (#43022)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:57:05 +08:00
shodoco e2fbc6a550 Stream a chunk once stream_interval tokens are unsent (#42643) 2026-10-08 10:12:37 -07:00
Eric Zhang 344b8a863b [Perf] Fuse short-convolution checkpoint tracking metadata (#43013) 2026-10-09 00:26:15 +08:00
Ke Bao 642e81fbb8 Fix DSA index host buffer cleanup (#43100) 2026-10-09 00:21:38 +08:00
Ke Bao 46ad9eb439 Fix K-only host buffer cleanup (#43101) 2026-10-09 00:21:27 +08:00
billishyahao 4ab4d90d78 [AMD][DCP 3/N] add aiter asm for kimi k3 target verify (#34537) 2026-10-08 09:13:03 -07:00
MickandMick Qian 3030e31ff8 [diffusion] refactor: deduplicate request extraction and ComfyUI test helpers (#42903)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 23:44:35 +08:00
黄孝君 9f5d871251 [NPU] Quote the arm64 Triton-ascend wheel URL in npu.Dockerfile (#43035) 2026-10-08 21:44:49 +08:00
943621c336 [sgl-router] Render DeepSeek-V4 through sglang-processor (#42665)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-08 20:16:20 +08:00
11b1fb21de [MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection (#38841)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:39:31 +08:00
6e6ccb26f2 [MiniMax-M3] Instantiate the decode block top-k in register buckets (#38615)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 19:33:17 +08:00
45abea0269 [rust-processor] DeepSeek-V4 parity with SGLang, plus a reusable parity harness and skill (#42664)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:35:30 +08:00
Shangming Cai 9578cb1c63 [CI] Don't run the full GPU suite for renderer-only Cargo.lock changes (#43025) 2026-10-08 18:25:06 +08:00
MickandMick Qian 7dd512b0e8 [diffusion] refactor: deduplicate FLUX positions and RoPE application (#43002)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 18:18:39 +08:00
Horace He 8ad24fb560 [Perf] Cache compiled ModelOpt exclusion patterns (#42927) 2026-10-08 17:10:25 +08:00
Liangsheng Yin 4b384df0c9 [CI] Keep the KL tests' LongBench cache outside the checkout and bound its download (#43068) 2026-10-08 01:17:31 -07:00
Liangsheng Yin 7b9c42cf49 [CI] Remove the unregistered chunked-prefill tests under test/manual (#43081) 2026-10-08 01:16:39 -07:00
Liangsheng Yin 89b5935793 [MemCache] Return only the loaded length from init_load_back; drop empty_device_indices (#43023) 2026-10-08 01:14:33 -07:00
Liangsheng Yin 5a4d01a3ae [MemCache] Return only the matched length from match_prefix; read KV indices off the node path (#42923) 2026-10-08 01:10:14 -07:00
Spandan Tiwari 2d26823f94 [AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token) (#42152) 2026-10-08 01:06:34 -07:00
Cheng Wan b3e6c87be3 [Refactor] Translate each KV loc once per iteration (#42753) 2026-10-08 00:58:28 -07:00
508ddc859a [unified-memory] Admit DFLASH and DSPARK with fused draft KV (#40492)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:56:21 -07:00
c2d491e38b [unified-memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts (#40491)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:55:54 -07:00
Michael 5afe4d6334 [AMD] Fix DSpark draft bucket tests for the raw in-graph metadata default (#42230) 2026-10-08 00:54:26 -07:00
8b80808366 [unified-memory] Fuse the draft model's KV into the target's page envelope (mechanism) (#37628)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:54:19 -07:00
08d30f2300 [unified-memory] Speculative decoding on the unified memory pool: translate every read and write path (#37627)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:52:21 -07:00
33883a5fa3 [AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950 (#41134)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 00:50:16 -07:00
b2cd65b8b0 [AMD] One-launch small-M MoE router for Qwen3.5 on gfx950 (#41133)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 00:47:52 -07:00
f795fe352d [Test] Add a docs drift check for server_arguments.mdx (#38520)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:37:22 -07:00
Chunan Zeng 785005d21b [DSpark] Mask uncommitted slots before commit-inject cache lookups (#42624) 2026-10-08 00:16:54 -07:00
Yuwei An 54b551df8e [Refactor][TCPCG] Bind live forward batches in BCG eager calls (2/9) (#42286) 2026-10-07 23:55:41 -07:00
MickandMick Qian 5c5f593110 [diffusion] refactor: share KL VAE encode/decode and tiling operations (#42841)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:43:19 +08:00
MickandMick Qian d1b751d958 [diffusion] refactor: inherit existing defaults instead of repeating overrides (#42963)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:42:21 +08:00
74308cf1d6 [diffusion] fix: wait for multimodal process teardown (#40765)
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:36:17 +08:00
4f320fb85a [diffusion] perf: FastH3 8-Step V2 native support, native Blackwell VSA and batched H3 VAE decode (#37662)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-08 14:35:27 +08:00
MickandMick Qian 57e9ff5403 [diffusion] refactor: share timestep retrieval across pipeline stages (#42967)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:33:39 +08:00
Liangsheng Yin 0f5bc4af4b [CI] Fix stale fakes in the NPU CUDA-graph backend unit test (#43056) 2026-10-07 23:04:48 -07:00
Mohammad Miadh Angkad ac656ee79a [CI] Re-enable B300 (#42959) 2026-10-07 22:03:16 -07:00
Jzz1943 b7b2975b57 fix(dsa): align FA3 metadata with unpadded queries (#42793) 2026-10-07 21:37:21 -07:00
hmalgewattaandAlex Nails 7327fb225f [DSV4] Guard V4.1 indexer SM100 check on non-CUDA devices (#42114)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-08 12:06:28 +08:00
Thomas Wang b72da1b817 [AMD] Update v4 amd cookbook 1008 (#43033) 2026-10-08 11:18:41 +08:00
AndyLi429andAndyLi429 89bea053ac [NPU] Add NPU cudagraph runner unit tests (#40404)
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com>
2026-10-08 11:18:38 +08:00
chilltongxandchilltongx e7530a3a26 [diffusion] fix: fix forward multimodal token type IDs in image encoding (#34494)
Co-authored-by: chilltongx <284668524+chilltongx@users.noreply.github.com>
2026-10-08 10:55:55 +08:00
WenhaoZhang b0d7254fbc [diffusion] CI: rebase LTX H100 timing references on recent CI runs (#42986) 2026-10-08 10:42:12 +08:00
Jiajun LiandClaude Opus 5.5 99463390f8 [Fix] MiMo-V2: accept checkpoints with the split attention layout (#40978)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 19:18:32 -07:00
c892301ff7 [Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities (#35594)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
2026-10-07 19:16:10 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 7b6b6ee25e [Docs] Cookbook: interactive SemiAnalysis InferenceX AgentX launches for GLM-5.2, DeepSeek-V4 and DeepSeek-V4.1 (#42726)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:33:40 +08:00
MickandMick Qian 21fa2251a0 [diffusion] refactor: share sigma schedules and causal cross-attention (#42848)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:29:33 +08:00
MickandMick Qian 28c08b02c3 [diffusion] refactor: deduplicate MOVA initialization (#42983)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:28:06 +08:00
MickandMick Qian 243f3dc35f [diffusion] refactor: deduplicate Qwen-VL inputs and component accuracy suites (#42860)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:27:22 +08:00
MickandMick Qian 2489bd1d2e [diffusion] refactor: share ModelOpt checkpoint conversion I/O (#42856)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:26:39 +08:00
MickandMick Qian 35608e48d1 [diffusion] refactor: reuse the single-process parallel test fixture (#42964)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:24:53 +08:00
ashwini rathi b91919ffe2 [XPU][diffusion] Re-baseline Wan2.1-T2V-1.3B TextEncodingStage on B60 (#42794) 2026-10-08 09:22:41 +08:00
MickandMick Qian 287b8b793f [diffusion] refactor: deduplicate VAE resampling and LTX tiling (#42866)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:22:11 +08:00
MickandMick Qian 0c00bffc6f [diffusion] refactor: deduplicate Qwen Image RoPE and SenseNova backbones (#42843)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:20:55 +08:00
MickandMick Qian 1e2f524726 [diffusion] refactor: share stacked encoder weight loading (#42988)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:16:03 +08:00
Liangsheng Yin 0427ea8032 [CI] Split the shared Cargo target dir per glibc (#43024) 2026-10-07 18:14:26 -07:00
MickandMick Qian d52ea5ee90 [diffusion] refactor: deduplicate realtime transport and LingBot tests (#42990)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:10:10 +08:00
Richard Wang 5cb7fb886c [Feature] support pplx-decider-v1.1 and update the pplx-decider cookbooks (#42645) 2026-10-08 09:03:50 +08:00
MickandClaude Opus 5.5 9101e895ea [diffusion] feat: encode the MP4 while the VAE decodes (MiniMax-H3, Wan) (#41819)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:01:19 +08:00
Cheng Wan 7760d5264a [Refactor] Use destination layouts for model checkpoint mappings (#42590) 2026-10-07 16:24:22 -07:00
Cheng Wan 5e12831cec [Refactor] Load auxiliary and expert checkpoints using owner partitions (#42589) 2026-10-07 16:24:03 -07:00
Cheng Wan 64c7f65eac [Refactor] Retain process groups in parallel linear layers (#42588) 2026-10-07 16:23:40 -07:00
Cheng Wan 267e34516c [Refactor] Select parallel groups for remaining model projections (#42587) 2026-10-07 16:22:57 -07:00
Cheng Wan dec5d9d6fc [Refactor] Let vision encoders select their parallel group (#42586) 2026-10-07 16:22:32 -07:00
Cheng Wan b83827ea02 [Fix] Quantize reloaded FP8 weights after native sharding (#42585) 2026-10-07 16:20:47 -07:00
Cheng Wan 59740a728f [Refactor] Let MLA projections select their parallel group (#42484) 2026-10-07 16:19:33 -07:00
Cheng Wan 961d7290fe [Refactor] Let model MLPs select their parallel group (#42468) 2026-10-07 16:18:14 -07:00
Cheng Wan 4c5130ac5a [Refactor] Let linear layers select their parallel group (#42426) 2026-10-07 16:15:26 -07:00
Liangsheng Yin 1066decce6 [Scheduler] Replace extend_range with extend_end derived from prefix_len (#42822) 2026-10-07 16:07:16 -07:00
Liangsheng Yin 5349d3e9d2 [Scheduler] Derive prefix KV indices from last_node at allocation; drop Req.prefix_indices (#42825) 2026-10-07 16:03:09 -07:00
Liangsheng Yin e9240b3243 [Scheduler] Track prefill progress as prefix_len (#42824) 2026-10-07 15:59:06 -07:00
Kevin MiandClaude Opus 5.5 79ef49cb6b [Docs] GLM-5.3-Flash: measured MI355X MXFP4 agentic recipe (#43015)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:49:49 -07:00
Liangsheng Yin 4bf8f28d15 [Scheduler] Fold match write-back into match_kv_cache (#42823) 2026-10-07 15:43:05 -07:00
Liangsheng Yin b1ffe5c16a [dLLM] Keep the request row across FDFO blocks (#42846) 2026-10-07 15:39:33 -07:00
Ping QiuandZhiqiang Xie 685d4d453d [HiCache] Add SeaweedFS L3 storage backend (#42399)
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-07 15:05:37 -07:00
WenhaoZhang 0b635266d4 [diffusion] CI: warm up CI cases at the request's quality level (#42958) 2026-10-07 23:26:36 +08:00
00b21d6415 [diffusion] fix: reset DiT cache state at the start of each request (#42437)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 23:25:49 +08:00
Mohammad Miadh AngkadandMohammad Angkad 44d10eab5e [GLM-V] Shard the vision tower over the attention-TP group under DP attention (#42931)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-07 23:17:06 +08:00
d9a0f1cb34 [diffusion] fix: fix hybrid SP+TP for I2V models (#41917)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 22:33:03 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 1e86143bf0 [Bugfix] Return None for empty reasoning_content outside K2 Horizon (#42962)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:02:40 +08:00
WenhaoZhang 492dbd0807 [CI] keep diffusion-only kernels and tests out of the srt stages (#42706) 2026-10-07 20:28:53 +08:00
Tianyang GuandClaude Fable 5.1 aa5551d9b6 [diffusion] fix: do not interleave comfy-kitchen MXFP8 scales twice (#38028)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 17:56:39 +08:00
YC Yen-Ching Tseng c389cbaf99 [AMD][diffusion] Route bf16 fuse_scale_shift to the native fallback on gfx1250 (#42859) 2026-10-07 02:37:43 -07:00
YC Yen-Ching Tseng 49d7d60dfa [AMD][diffusion] VAE attention to the math SDPA backend on gfx1250 (#42858) 2026-10-07 02:32:02 -07:00
Yihao WangandClaude Opus 5.5 11972e5207 [deps] bump transformers to 5.19.0 (#42784)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 17:04:08 +08:00
Rain Jiang c52dcc5a01 The sgalng api specs (#41585) 2026-10-07 01:55:01 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 ba77d30256 [FA4] Keep stream last in the SM100 MLA kernel so disk-cached objects load (#42926)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:58:55 +08:00
Khoa Pham e23093077e [DSv4] Fix shared index cache accessor page size (#42832) 2026-10-07 00:53:17 -07:00
jiaryang 1c42ad3679 [ROCm] Skip the tilelang act_quant in the DSA indexer on gfx1250 (#42747) 2026-10-07 00:16:46 -07:00
Shuwen WangandClaude Opus 5 2889158792 [MemCache] test: require every ComponentData field to be classified by each split hook (#42518)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-07 14:53:34 +08:00
paulzhang-tm c721602ad2 Cover SWA admission after declined HiCache restores (#39839) 2026-10-07 14:51:45 +08:00
Hyogeun Oh (오효근)andmickqian 1d019ce1c8 [diffusion] fix: honor the lazy transformer loader contract (#15215)
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
2026-10-07 14:23:00 +08:00
Mark b0dff4009f [diffusion] cli: restore gated Diffusers model detection (#34903) 2026-10-07 14:21:37 +08:00
Mohammad Miadh AngkadandMohammad Angkad f47d89564d [KDA] Default GLM-5-Next spec verify to the CuTe DSL KDA kernel (#42905)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 22:31:41 -07:00
AMD-yanfeiwangandThomas Wang 9ddbba50dd [AMD][DSV4] feat: enable PCP with fp8 unified_kv on gfx950 (#39923)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-06 22:09:18 -07:00
Seas0andMick Qian 43b4abd857 [diffusion] fix: validate SageAttention 2 head sizes without overriding component policy (#41529)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 12:39:54 +08:00
Xinyi Song 168458fd90 [AMD][V4.1] Fix wo_b input flatten for packed fp8 activations on gfx950 (#42796) 2026-10-06 21:35:51 -07:00
Khoa Pham 9e26600f3a [DSA] Clarify pooled-page naming and ownership (#42671) 2026-10-06 21:05:29 -07:00
CuzMi bde01f8dc8 [Diffusion] Fuse Anima AdaLN modulation and cache the FFI callable (#42608) 2026-10-07 12:01:59 +08:00
Tuan Nguyen GiaandCursor c0df52be2d [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path (#34540)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-07 03:20:40 +00:00
Yi (Vincent) ZhongandClaude Opus 5.5 012cd3858b [Fix] Make the fused KDA beta sigmoid bit-identical to torch.sigmoid (#42611)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 20:04:59 -07:00
Thomas Wang 9a21d82958 [AMD] Update v4.1 amd cookbook 1006 (#42847) 2026-10-06 19:58:58 -07:00
Jay Zenith 35f18848db [diffusion] fix: validate output_quality and fix return type annotation (#19530) 2026-10-07 10:53:29 +08:00
Yash AkhauriandBBuf 9c8fc6a8e4 [Bugfix] Preserve empty K2 Horizon reasoning in chat responses (#42623)
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-07 10:44:55 +08:00
d7190b0122 [Perf] Accelerate DiffusionGemma with FA4 and CUDA graphs on Blackwell (#42239)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-07 10:31:00 +08:00
2f6e0ec45d [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-07 09:50:34 +08:00
7a4d6dfd2f [diffusion] feat: name the quality tiers after what they guarantee (#42370)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-07 09:49:30 +08:00
Lianmin Zheng 8f1790f595 Add Rust code owners and grant shodoco CI permissions (#42826) 2026-10-06 18:37:54 -07:00
660b17e8dc [AMD] Let AITER unified verify run without host seq_lens (#42635)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com>
2026-10-06 18:27:27 -07:00
Eddie A. Tsai e13933c4fd [diffusion] docs: add gfx1151 Strix Halo serving guidance (#41670) 2026-10-07 09:14:07 +08:00
Michael 3e914e69d5 [AMD] ci: nightly-test the miles ROCm 10 MI30X image on MI300 (#42771) 2026-10-06 18:05:09 -07:00
WenhaoZhang 6da08fd7cf [diffusion] CI: fix qwen cache-dit step 10/49 perf baselines (#42495) 2026-10-07 08:51:12 +08:00
MickandMick Qian 1351f32bb4 [diffusion] refactor: simplify Kandinsky 6 models, SR sampling and tests (#42768)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 08:48:16 +08:00
f1065423d3 [PD] Stop reserving unallocated mamba spec-verify scratch in prefill KV sizing (#42649)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-06 17:16:22 -07:00
Baizhou Zhang f497b626b6 build: use one DeepEP ref for implementation and packaging (#42827) 2026-10-06 17:04:28 -07:00
Duyi-Wang 66dd4e15b9 [AMD][Docs] Add MI355X DeepSeek-V4 Pro PD recipes (#42757) 2026-10-06 16:58:40 -07:00
08aad0c196 [HiCache] Manage buffer-mode backups per storage pool (#42633)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-06 16:45:05 -07:00
jain-riaandishandhanani fd62ee84f7 feat(grpc): expose follower metadata and node-local KV sources (#39659)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-06 16:39:01 -07:00
Chunan ZengandCheng Wan 90aa1cb0fb [Scheduler] Gather prefill-delayer queue timeout across ranks (#42625)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-06 16:15:34 -07:00
Jan BernlöhrandPo-Han Huang 0a0fe001f1 fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends (#41765)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-06 16:02:45 -07:00
6b737fd4c6 Bind the MessageQueue remote socket atomically (#42695)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-06 15:50:59 -07:00
Trevor Morris 44a2558d40 [NVIDIA] Fix nightly-cu134 build and update (#42661) 2026-10-06 14:52:42 -07:00
264d1c2015 Back the DFlash-family draft KV pool to the post-capture token count (#42651)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-06 14:23:18 -07:00
4506927635 [Metrics] Add per-rank DP attention imbalance metrics (#42755)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-06 14:20:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 38bdf25f8d [Docker] Use uv-managed Python 3.12 and uv for all installs in the CUDA image (#42612)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 14:10:09 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 3eec178e48 [Spec][MegaMoE] Let the speculative draft choose its own W4A4 MXFP4 MegaMoE MMA type (#42022)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:09:57 -07:00
Liangsheng Yin 12004043b8 [Metrics] Read scheduled prefill KV tokens from batch prefix_lens (#42797) 2026-10-06 13:11:29 -07:00
a34cba3c62 [HiCache] Resolve write_back to write_through under buffer_only host memory (#42754)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-06 11:02:23 -07:00
0f57a39152 [rust-processor] Gate render, tokenizer and parser behind cargo features (#42663)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:58:38 -07:00
Byron Hsu abed685de1 [PD] Fix crash for allocation when mamba + pd + decode radix cache (#41927) 2026-10-06 09:36:31 -07:00
MichaelandClaude 209e190ee5 [AMD] ci: build the miles ROCm 10 MI30X image nightly (#41605)
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-07 00:00:11 +08:00
Emil BogomolovandEmil Bogomolov e17935820a [diffusion] fix: fix width/height silently dropped on multipart /v1/videos requests (#35108)
Co-authored-by: Emil Bogomolov <zetyquickly@googlemail.com>
2026-10-06 23:31:31 +08:00
WenhaoZhangandMick 2579875b5d [diffusion] feat: support loading community ComfyUI-GGUF Qwen-Image-2.1 transformers (#42556)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-06 23:30:48 +08:00
49bafb58cd perf(mla): use native SATFINITE FP8 conversion in MLA KV/Q preparation (#42659)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 22:47:47 +08:00
Xiaoshuai Zhangandjet 679d2851ba fix(mimo-vl): keep multimodal features when capturing aux hidden states and correct vision preprocessing (#40888)
Co-authored-by: jet <dev@jetd.one>
2026-10-06 22:15:14 +08:00
16a23a672b [diffusion] model: support Kandinsky 6 (#42743)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Lev Novitskiy <57654885+leffff@users.noreply.github.com>
Co-authored-by: Kirill <kykozlov@gmail.com>
Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
2026-10-06 22:13:29 +08:00
730f1f3e5b [DeepSeek-V4.1] Unify mHC into one state machine and drop medium-batch fusion variants (#42245)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-06 21:01:42 +08:00
WenhaoZhang 2a5f45d3d4 [diffusion] fix: skip zero blocks when hashing large conditioning inputs (#42748) 2026-10-06 20:43:08 +08:00
WMCandjacky.cheng ba94488b42 [diffusion] feat: default Qwen-Image 2.1 to VAE tiling on gfx1151 (#41655)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-06 20:20:58 +08:00
Shangming Cai 1291f3175d [rust-processor] Fix chat-template precedence wording in the processor README (#42756) 2026-10-06 19:10:26 +08:00
Shangming CaiandClaude Fable 5.1 28525b43c4 [sgl-router] Document and reject a zero request timeout (#42751)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 19:07:37 +08:00
cf19ae6d04 [rust-processor] Split sglang-processor into tokenizer, render and parser (#42662)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:59:02 +08:00
Mohammad Miadh AngkadandMohammad Angkad 45db430530 [DSA] Fix pooled page table refresh for target verify CUDA graph replay (#42750)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 18:57:52 +08:00
2167cf90c3 [sgl-router] Bound streaming time-to-headers by --request-timeout-secs (retry 5/5) (#42632)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:35:11 +08:00
YC Yen-Ching Tseng f3b5a28f43 [AMD] M3-mxfp8 Nightly Test Update (#42605) 2026-10-06 02:22:53 -07:00
YC Yen-Ching Tseng 542addad32 [AMD] M3-MXFP4 Nightly Test (#42602) 2026-10-06 02:22:01 -07:00
662879e495 [sgl-router] Back off between retry attempts (retry 4/5) (#42631)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-06 17:06:45 +08:00
Gawr Gura 0721 b563bbd597 [Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP (#40671) 2026-10-06 01:58:02 -07:00
e49e971131 [mem_cache] Keep prefix_indices current for chunked requests kept out of the tree (#42467)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-06 01:40:35 -07:00
bb4d0a13c1 [sgl-router] Retry failed dispatches on another worker (retry 3/5) (#42630)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-06 16:28:52 +08:00
Jzz1943 1be3013118 [diffusion] feat: support Ideogram TurboTime LoRA (#30487) 2026-10-06 16:15:21 +08:00
Max Kong 01345197af [diffusion] chore: harden OpenAI upload filename handling (#17270)
=
2026-10-06 16:14:39 +08:00
Jinbao ChenandMick Qian d0494af08b [diffusion] security: pin multimodal-gen ZMQ scheduler ingress to loopback on wildcard --host (#36854)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 16:05:17 +08:00
Liangsheng Yin c7f3ae1dd5 [mem_cache] Hold a request's tree lock as one TreeLock (#42686) 2026-10-06 00:59:38 -07:00
05340e9ecb [sgl-router] Make one request dispatchable more than once (retry 2/5) (#42629)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:24:02 +08:00
c683190b40 [PD] Wait for prefill completion before Mooncake early KV transfer (#41395)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-06 15:17:12 +08:00
8c273d6813 [sgl-router] Let selection skip workers a request already failed on (retry 1/5) (#42628)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:10:51 +08:00
Rumit Desai 33ce0943e9 [Bugfix] Guard merge_state sgl_kernel import on non-CUDA platforms (#42682) 2026-10-06 14:19:20 +08:00
Joe 40fb5935c0 Allow platforms to declare Mamba extra-buffer support (#36410) 2026-10-06 14:18:49 +08:00
Liangsheng Yin 9a09b48da7 [Fix] Add maybe_hand_to_session to the one-batch benchmark TreeCacheNamespace (#42723) 2026-10-05 23:09:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 60878c3fba [CI] Temporarily disable B300 (#42704)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-05 22:27:54 -07:00
b524de2de6 [diffusion] optimization: release the allocator cache when the server idles not per request (#41718)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 12:27:08 +08:00
e446698cdd [SM120&90] Add CUDA fused Triton sparse-MLA prefill backend for DSA (#32779)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-05 21:08:05 -07:00
1e1bdeedbb [Perf] Improve GLM-5.3-Flash DFlash decoding on B300 (#42489)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 12:05:37 +08:00
Mohammad Miadh AngkadandMohammad Angkad fcbbefa82a [GLM-5.3-Flash] Autotune MoE kernels at prefill sizes (#42693)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-05 21:04:54 -07:00
Yuwei An 620c76f25d [DSV4.1] Add opt-in TRT-LLM sparse attention support (#41603) 2026-10-06 12:02:10 +08:00
Zhang, Jiejing c76faa1ff2 docs(cookbook): enable Top-K v2 for GLM-5.2 MI355X MTP (#42613) 2026-10-06 11:33:24 +08:00
Ke Bao ed75fe67f1 Declare hybrid Mamba indexer host pool (#40915) 2026-10-06 10:18:36 +08:00
CuzMi 67959f80ab [diffusion] optimization: fuse Anima FP32 split-half Q/K RoPE (#42609) 2026-10-06 10:18:09 +08:00
Thomas Wang 1abd356595 [AMD] Add engram host/device address mapping (#42552) 2026-10-05 19:12:12 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 f2f0be731c [Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings (#42167)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:40:30 +08:00
MickandMick Qian 21c9bbdf2c [diffusion] fix: fix concurrent server perf dump writes across ranks (#41541)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 09:29:51 +08:00
Yash AkhauriandXiaoyu Zhang 36b216e2ee [Docs] Refresh K2 Horizon checkpoint revisions (#42130)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 09:19:04 +08:00
5d385007b3 [diffusion] kernels: take the Wan VAE decoder's remaining first-decode work off the request path (#41830)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:05:29 +08:00
3120bb87a2 [diffusion] fix: fix the OOM hint's placement advice and add a 24 GB Wan2.1 recipe (#41723)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:01:02 +08:00
Liangsheng Yin ad7a9b53cc [mem_cache] Make release_kv_cache's final checkpoint an explicit checkpoint flag (#42270) 2026-10-05 17:32:07 -07:00
Po-Han Huang (NVIDIA) 62af5f56a5 [CI] Drop the redundant and broken tc_piecewise tests (#42571) 2026-10-05 17:29:32 -07:00
Liangsheng Yin c01d59eb90 [Session] Let a streaming session own its KV record and tree lock from the first turn (#42537) 2026-10-05 17:28:09 -07:00
Liangsheng Yin f3fe753475 [misc] Rename SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA (#42657) 2026-10-05 17:24:15 -07:00
Sage 4409d2fd22 [ci] add renderer automatic labeler (#40923)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-05 17:12:51 -07:00
Yuwei An 18ec2ecce3 [Refactor][TCPCG] Relocate shared graph tensor and DSA head-gate helpers (1/9) (#42285) 2026-10-05 17:06:52 -07:00
ADiL 968726f3ce [HiCache] Route --file-storage-path to the file storage backend (#33883)
Signed-off-by: adlashab <adlashab@amd.com>
2026-10-05 15:33:49 -07:00
61ba98443d [DSA] k-pool: 256-token logical page so index page id = logical page id (#42178)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-10-05 15:02:44 -07:00
c765f8818a [DFlash2] Keep the greedy selector walk in range for all-NaN score rows (#41737)
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 14:29:13 -07:00
maocheng23 57826a62ea [Speculative] Add opt-in block verification (#42297) 2026-10-05 11:49:23 -07:00
70f0b7351e [sgl-router] Improved policy balanced mode (#42545)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-05 09:48:37 -07:00
debanshd efb62ce269 [diffusion] fix: resolve QKV layout inference for MiniMax-H3 checkpoints (#40158)
Signed-off-by: Debanshu Das <debanshu@google.com>
2026-10-05 23:01:37 +08:00
150f568900 [diffusion] kernels: stop specializing on per-request sequence lengths (#41710)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 22:21:01 +08:00
a977e3b9d5 [diffusion] optimization: stream an oversized DiT in auto mode instead of OOMing (#41721)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:58:27 +08:00
ad3e80d4de [diffusion] chore: warm the image writer and the HTTP route models at startup (#41833)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:58:00 +08:00
c83c84b422 [diffusion] chore: keep the DiT off component offload under explicit multi-GPU fsdp (#41722)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:56:52 +08:00
wxy 23bf107469 [diffusion] optimization: cache LTX-2 RoPE coords to avoid per-step recompute (#22441) 2026-10-05 21:21:36 +08:00
Shiwen Cheng 91f9bf8d17 [diffusion] fix: fix scheduler host not working when ipv6 in multi modal gen (#22813) 2026-10-05 21:14:09 +08:00
liukeandMick f20f8e120d [diffusion] model: restore folded H3 GGUF patch embedding (#39523)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-05 21:10:06 +08:00
MickandMick Qian 83a6e1d39b [diffusion] docs: keep model compatibility details in cookbook recipes (#42227)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 20:02:43 +08:00
Akshat Anandandcipheraxat 860ebe720e [diffusion] fix: skip warmup preferred preload when it would OOM (#40761)
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
2026-10-05 19:59:35 +08:00
aiyueqiandMick 8663e3b670 [diffusion] optimization: avoid full-video clone in VAE post-processing for MiniMax-H3 (#40095)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-05 19:56:11 +08:00
d349e672bc [diffusion] feat: recycle warmup-only entries of conditioning-cache at the first served store (#41832)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:55:30 +08:00
7352073ec6 [diffusion] feat: support --async-output-save to overlap output saving with the next request (#37549)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 19:53:38 +08:00
WenhaoZhangandMick Qian f048d5aa4b [diffusion] feat: add community LoRA recipes and Kohya mapping (#35857)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 18:22:56 +08:00
08fa37e48d [sgl-router] Add reorg bucket config with per-group admission (#42429)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-05 18:17:32 +08:00
Shangming CaiandClaude Opus 5.5 ab4f8a44f4 [sgl-router] Point the design doc at the shared engine ranking (#42572)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:16:42 +08:00
a96b146491 [sgl-router] Move pressure and prefix signals out of legacy policies (#42428)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:10:29 +08:00
mrusanovskyandPo-Han Huang 26a1377c50 [Spec] LiLiCorr: named head MLPs and quantized head linears (#42057)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-05 02:43:59 -07:00
WenhaoZhangandMick Qian 284cda01fb [diffusion] optimization: fuse the video VAE decoder's RMSNorm and QK RoPE for MiniMax-H3 (#41906)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 17:30:39 +08:00
0263fdacb2 [diffusion] fix: keep generic warmup valid for step-floored models and stop reporting a failed warmup as warm (#41641)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:23:06 +08:00
elvischenvandClaude Opus 5.5 55dcf3254f Fix XQA draft extend with FP8 KV cache in trtllm_mha (#41656)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 01:37:54 -07:00
Xiaoyu Zhang abd6c374c7 [Perf] Optimize MiMo local audio attention on Hopper (#42101) 2026-10-05 16:14:56 +08:00
Michael b3f967e809 Revert "Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)"" (#42558) 2026-10-05 16:12:21 +08:00
Aleksi Vesantoandjacky.cheng 9f7d38ffd3 [diffusion] attention: Allow disabling sequence masking in SP (#33855)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-05 15:22:53 +08:00
Aleksi Vesantoandjacky.cheng c41b2d26e5 [diffusion][ROCm][Perf]: Set gfx942 AITER FMHA rounding mode to rtz instead of rtna (#28650)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-05 15:21:04 +08:00
MickandMick Qian dd2ffdb680 [diffusion] nightly: keep nightly regression baselines on a comparable methodology (#42226)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 15:19:21 +08:00
438d9d2074 [Feature] DeepEP v2: expanded (do_expand=True) prefill dispatch (#37261)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-05 00:02:10 -07:00
cafb37a10d [JIT] Fuse FP8 KV-cache quantization into the prefix-valid commit kernel (#40785)
Co-authored-by: Jayden Yu <jaydenyu@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-05 14:47:54 +08:00
WenhaoZhangandMick Qian f218798006 [diffusion] CI: relax the load latency check and extend the 2-GPU retry deadline (#42505)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 14:43:27 +08:00
520d6d0229 [DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs (#41659)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-05 14:33:37 +08:00
Liangsheng Yin 503b898540 [Session] Count streaming session KV by owner in pool accounting (#42539) 2026-10-04 22:15:57 -07:00
Michael 456d533e61 Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)" (#42541) 2026-10-04 21:59:12 -07:00
Liangsheng Yin 1818db4e54 [CI] Drop the echoed command from /rerun-test result replies (#42543) 2026-10-04 21:50:53 -07:00
ashwini rathi b1bbd74f28 [XPU][CI] Disable test_xpu_graph until tc_piecewise is removed (#41634) (#42524) 2026-10-05 11:09:26 +08:00
Xiaoyu Zhang f385390be5 [Refactor] Simplify GDN, vision RoPE and MLA kernel dispatch (#42506) 2026-10-05 09:37:53 +08:00
734cf3cf3b [PD] Reload NIXL peer metadata after the peer is invalidated (#41691)
Co-authored-by: agent <agent@local>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-05 00:49:26 +00:00
Yuwei An f70e8c68fc [Dsv4.1] Bounded replay with sparse mla path (#42273) 2026-10-04 17:19:14 -07:00
Eric.Chin.AMDandBingxu Chen b792228b35 [ROCm] topk v2: split one long row across blocks, the CDNA cluster-path equivalent (#39931)
Co-authored-by: Bingxu Chen <bingxche@amd.com>
2026-10-04 16:54:37 -07:00
Cheng Wan 1f299e9f2f [Refactor] Build layer stacks in order with append_stages (#42481) 2026-10-04 16:40:24 -07:00
Cheng Wan a99567ab74 [Fix] GigaChat 3.5: build the stage boundaries once (#42480) 2026-10-04 16:39:34 -07:00
Cheng Wan 5c8fb9e938 [Fix] Offer the fused MoE finalize all-reduce only to a block that owes the sum (#42479) 2026-10-04 16:39:06 -07:00
Cheng Wan f07c1a398e [Refactor] Build the Qwen4 experimental decoders from stage boundaries (#42478) 2026-10-04 16:38:36 -07:00
Cheng Wan 92d60351e2 [Refactor] Build the Hunyuan V4 decoder from stage boundaries (#42477) 2026-10-04 16:38:03 -07:00
Lianmin Zhengandjiayisuse f5ccc4d7d9 [Kernel] Use explicit reciprocal scaling for recurrent Q/K normalization (#42486)
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-04 15:30:03 -07:00
Tarang Khanna d475b5a25c [weight_cache] Coordinate daemon readiness across ranks (#38828) 2026-10-04 15:20:13 -07:00
Aditya Sharma caf96324fc [Apple Silicon] Add an Apple Silicon (MPS) platform to the standard Torch model runner (#36780) 2026-10-04 15:03:40 -07:00
SuperSongandCheng Wan bab04cd791 fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots (#42072)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 12:51:22 -07:00
Mohammad Miadh AngkadandMohammad Angkad fea3c088a1 [CI] Temporarily disable GB300 (#42463)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 12:24:04 -07:00
Zhang, Jiejingandkyle-256 aa0ce20201 [ROCm] GLM-5.2 decode path: decode-shaped MoE/MLA tiles, split speculative softmax, and bf16 GEMM routing (#41725)
Co-authored-by: kyle-256 <Kyle.Zhao@amd.com>
2026-10-04 11:19:37 -07:00
Xiaoyu Zhang 7a719e9a65 [AMD] Fix RoPE cache dtype and diffusion CI failures (#42494) 2026-10-05 00:36:26 +08:00
billishyahao 0343bc1483 [AMD][Docs] fix mori io and umbp playground (#42512) 2026-10-04 09:28:11 -07:00
Shuwen Wang 137c084000 [HiCache] refactor: retire in-flight storage prefetches through one helper (#41453) 2026-10-05 00:13:07 +08:00
WenhaoZhang affa261e3d [diffusion] feat: enable cuda graphs for observation encoding and denoising steps for action models (#42171) 2026-10-04 22:22:08 +08:00
WenhaoZhang 279d3f3888 [diffusion] feat: load ComfyUI/ai-toolkit fused gate_up LoRAs and diffusers metadata alpha for qwen-image-2.1 (#42174) 2026-10-04 22:21:13 +08:00
WenhaoZhang 44fac689f5 [diffusion] optimization: decode json pixel lists to numpy at the entrypoint (#41895) 2026-10-04 22:20:20 +08:00
Michael ab9d8f029c [AMD][DI][CI] mi355x spur: skip RDMA ports that are not PORT_ACTIVE; dump driver log on failure (#41405) 2026-10-04 21:23:54 +08:00
Shuwen Wang 35f3c96ff4 [CI] Fix bootstrap sender registration test on main (#42487) 2026-10-04 18:45:31 +08:00
Shangming CaiandClaude Opus 5 148da8e77d [PD] Validate decode state layout once at registration (#42051)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 18:32:30 +08:00
Xiaoyu Zhang bc5a055dd7 [diffusion] move kernel validation into launchers and simplify dispatch (#42391) 2026-10-04 18:23:29 +08:00
Michael 29bcb74a89 [AMD][CI] Fix ROCm kernel wheel sources: drop eagle_utils, add DSV4 kernels (#42461) 2026-10-04 17:54:28 +08:00
SuperSongandCheng Wan 6cec8f98ce [Bugfix] Fix unified-memory compaction gates and pending page reuse (#39982)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 02:48:08 -07:00
SuperSongandCheng Wan 1d02a36bb7 fix(inkling): translate unified-memory checkpoint destinations to physical slots (#38229)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 02:45:45 -07:00
WenhaoZhangandClaude Opus 5.5 1e490772e5 [diffusion] fix: pin the JoyAI-Echo overlay's source revision (#42471)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 17:19:09 +08:00
ba68306256 [diffusion] feat: let requests choose the video libx264 preset (#41797)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 16:22:36 +08:00
Tokha233andWenhaoZhang fbce0d9478 [diffusion] fix: load serialized h3 INT8 in ComfyUI integrated mode (#42121)
Signed-off-by: Tokha233 <61346912+Tokha233@users.noreply.github.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
2026-10-04 16:08:56 +08:00
Mohammad Miadh AngkadandMohammad Angkad 0bdd0cc8dc [Deps] Upgrade the CUDA PyTorch stack to 2.14 (#38641)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 00:48:02 -07:00
Kotthagattu Meher Sai 9631f6f2d0 [XPU] Enable fused QK-norm + RoPE for Qwen3-MoE (#40190) 2026-10-04 15:45:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7eb628612c [Fix] Publish the parallel config the CLIP attention test reads (#42460)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 00:44:51 -07:00
Baizhou Zhang 55c5415c17 [CP] Support scattered interleave CP inputs with expert parallelism (#42041) 2026-10-04 00:43:31 -07:00
937c0a6cfe Fix unified HiCache physical transfers (#39479)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-04 00:38:56 -07:00
billishyahao 6cc661f223 [AMD][DCP 2/N] Enable aiter asm ps for kimi k3 prefill (#34438) 2026-10-03 23:34:45 -07:00
Liangsheng Yin 048b2a07f3 [Session] Release aborted streaming turns through the normal release path (#42330) 2026-10-03 22:30:25 -07:00
Mohammad Miadh AngkadandMohammad Angkad 1315d171a0 [Diffusion][CI] Add missing B200 Cirrascale runner E2E baselines (#42329)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:10:32 -07:00
Brayden ZhongandMohammad Angkad a3275295c4 [MoE] Use FlashInfer A2A for prefill instead of AG+RS (#39818)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:09:00 -07:00
Mohammad Miadh AngkadandMohammad Angkad c12bc635ca [CI] Install only userspace libgdrapi for GDRCopy (#42233)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:05:35 -07:00
Mohammad Miadh AngkadandMohammad Angkad 0e6d7eba5f [Fix] Count attention-DP ranks in one-batch bench running cap (#42331)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:04:27 -07:00
Baizhou Zhang fccc61c1c5 [CP] Support DP x TP x CP for attention with interleave CP strategy (#42038) 2026-10-03 20:57:36 -07:00
Aurick Qiao f24db66729 [PD] Publish prefill DP rank whenever forced lookup is enabled (#41786) 2026-10-04 11:46:20 +08:00
metamergebotandmetamergebot 169e6223ef [PD] Drop the removed MooncakeKVSender args from test_prefill_complete_integration (#42454)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-03 20:29:06 -07:00
4611ea365e [HiCache]: Fix host lock ownership across radix splits (#38480)
Signed-off-by: Lukas Humbel <lhumbel@nvidia.com>
Co-authored-by: Jialin Ouyang <jialino@meta.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-04 11:16:04 +08:00
ff219e159e [AMD] MiniMax-M3: fuse sparse QK norm, RoPE and cache writes with AITER (#35357)
Co-authored-by: zijiec <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-03 19:49:53 -07:00
sglang-botandsglang-bot 8afef58af4 docs: sync LMSYS SGLang blog cards (#41296)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-04 02:26:05 +00:00
5606fb8592 [Fix] HiCache: carry registered Mamba slot side states (Qwen4-Exp PLE) through the host tier (#39862)
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
2026-10-04 10:15:59 +08:00
Michael f2385314b2 [AMD] Add DeepSeek-V4.1-Flash MI35x nightly accuracy test (#41476) 2026-10-03 18:16:44 -07:00
Xinyi Songandkk 663984b5a1 [AMD] Keep aiter's tuned GEMM for the DeepSeek-V4 compressors (#41931)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
2026-10-03 18:11:37 -07:00
trebladev 295a53c90e [diffusion] optimization: fuse Flux3 rowwise FP8 quantization with triton (#41671) 2026-10-04 09:01:17 +08:00
71dc292b97 [AMD] Use AITER ASM prefill for MiniMax-M3 HD128 attention (#41707)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-03 17:54:49 -07:00
4ab720e655 [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails (#42420)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com>
2026-10-03 16:01:32 -07:00
metamergebotandmerrymercy 4301be7bdb Pick the default nccl_port below the kernel ephemeral range (#42405)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
2026-10-03 16:00:53 -07:00
Cheng Wan 10edb05e72 [Refactor] Let the remaining placement consumers read the parallel context (#42348) 2026-10-03 14:59:32 -07:00
07064fd4a4 [sgl-router] Credit routed prompts before their KV events arrive (#42276)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-03 14:57:49 -07:00
Cheng Wan b2aa993cb8 [Fix] Skip a draft's decode recapture when it owns no decode graph (#42347) 2026-10-03 14:54:12 -07:00
Cheng Wan 1a18de4b29 [Fix] Load a draft's tensor weight update from its deployment TP rank (#42346) 2026-10-03 14:52:25 -07:00
sglang-botandsglang-bot 061e712bab chore: bump sgl-kernel version to 0.4.9 (#42427)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-03 14:29:04 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3122b9c995 Update AOT kernels for Torch 2.14 (#42366)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 14:26:34 -07:00
Liangsheng Yin df53c978f1 [mem_cache] Run mamba models on UnifiedRadixCache when the radix cache is disabled (#42354) 2026-10-03 14:11:13 -07:00
Liangsheng Yin 3e9a120add [Fix] Make Hf3fsMockClient reads and writes thread-safe with pread/pwrite (#42425) 2026-10-03 14:02:24 -07:00
8bf780dda5 feat(metrics): expose deferred decode KV release metrics (#41128)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-04 04:52:21 +08:00
Yanbin Jiang 749a89abd6 [LoRA] Reorganize kernels and add CODEOWNERS (#42299) 2026-10-03 13:43:40 -07:00
fd5e68f99e [sgl-router] Repair KV-event sequence gaps from the engine's replay socket (#42275)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-04 04:43:17 +08:00
aaf7ebe077 Advertise the KV-event replay endpoint in /server_info (#42274)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 04:42:33 +08:00
AMD-yanfeiwang 3dc7f4b816 fix(cuda-graph): remove unnecessary CP batch alignment (#40543) 2026-10-03 12:14:28 -07:00
AMD-yanfeiwang e83c95b650 [AMD] Fix AITER weight slicing for CP decode TP (#39950) 2026-10-03 12:09:08 -07:00
AMD-yanfeiwang f327f92424 [AMD] Port CP V2 to the DeepSeek-V4 HIP backend (#34200) 2026-10-03 12:08:17 -07:00
6acf2f4736 [PD] Keep ingesting requests while a prefill forward result is pending (#42035)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-03 12:02:21 -07:00
b016ca406f Support post-capture KV sizing for the unified hybrid-SWA pool (#41961)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-03 09:54:20 -07:00
2bae12b9b3 [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert (#42264)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-03 09:45:56 -07:00
5b5d721239 [PD] Add opt-in prefill-complete decode KV allocation (#40703)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-03 21:03:09 +08:00
5916999afb [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-03 19:31:05 +08:00
d75d5b33f7 [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-03 18:25:22 +08:00
DarkSharpnessandClaude Opus 5 379ec90fb2 [DSv4.1] Fold q_rope_store into fused_q_norm_rope (#41657)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 18:24:41 +08:00
Cheng Wan 7be5e3473c [Refactor] Drop the reduction-skip mechanisms stage boundaries no longer use (#42312) 2026-10-03 02:21:55 -07:00
Cheng Wan 71b04e02cf [Refactor] Build the ZAYA1, IQuest-Q1 and Gemma 4 decoders from stage boundaries (#42311) 2026-10-03 02:21:34 -07:00
Cheng Wan f85c2c4923 [Fix] GigaChat 3.5: apply the sandwich norms to the complete sums (#42310) 2026-10-03 02:21:10 -07:00
Cheng Wan 6faccf3e84 [Refactor] Build the GLM-4, GLM-Image and Granite MoE hybrid decoders from stage boundaries (#42309) 2026-10-03 02:20:40 -07:00
Cheng Wan e4554fd5e5 [Refactor] Build the Llama and Nemotron-NAS decoders from stage boundaries (#42308) 2026-10-03 02:20:18 -07:00
Cheng Wan 740a4d5955 [Refactor] Build the Qwen2 decoders from stage boundaries (#42307) 2026-10-03 02:19:56 -07:00
Cheng Wan 2c739b225b [Fix] Jet-Nemotron build and Granite MoE hybrid final norm (#42306) 2026-10-03 02:19:33 -07:00
Cheng Wan b7fc516986 [Fix] EXAONE MoE under DP attention and DeepEP (#42305) 2026-10-03 02:19:19 -07:00
Cheng Wan 2fa17a6467 [Refactor] Build the ERNIE 4.5 VL MoE and EXAONE MoE decoders from stage boundaries (#42304) 2026-10-03 02:18:56 -07:00
Cheng Wan f89ead192f [Fix] EXAONE and ERNIE 4.5 VL MoE architecture, backend and PP issues (#42303) 2026-10-03 02:18:33 -07:00
Cheng Wan be20e7453c [Fix] Step-3.5 DeepEP routed scaling and Sarvam shared expert under dense TP1 (#42302) 2026-10-03 02:18:11 -07:00
Cheng Wan b157d548bb [Refactor] Let stage boundaries complete every stage-output sum (#42301) 2026-10-03 02:17:48 -07:00
Cheng Wan d5a8d76bf5 [Refactor] Drop TBO op methods that no strategy ever schedules (#42300) 2026-10-03 02:17:26 -07:00
Liangsheng Yin 122b59152c [mem_cache] Replace is_chunk_cache / is_tree_cache with supports_prefix_sharing (#42362) 2026-10-03 02:07:55 -07:00
Banruo Liu 8293f9af54 [PD] Centralize drain-aware abort acknowledgements (#37077) 2026-10-03 16:51:11 +08:00
saatwiknagpal 1093c501df [diffusion] optimization: route subBlock sparse attention in head chunks (#41985) 2026-10-03 15:54:54 +08:00
Yanbin Jiang 254aa41f78 [Fix] Guard DeepSeek NVFP4 shared-expert fusion for LoRA and FP4 backends (#42203) 2026-10-03 00:37:57 -07:00
Tokha233 ce617d410b [diffusion] fix: fix native FP8 format handling for FLUX 3 row-wise linears (#42255) 2026-10-03 15:23:41 +08:00
Mohammad Miadh AngkadandMohammad Angkad 00bcc25f6c [HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag (#42215)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 22:27:57 -07:00
Jimmy Shong 0072e2cc08 [Fix] Validate per-layer rope_parameters on transformers 5.17 (Laguna-S/XS-2.1 nightly) (#42214) 2026-10-02 21:49:45 -07:00
Jacob0226andThomas Wang af1bef3eaf [AMD] GLM-5.3-Flash: fuse shared expert and KDA projections on Quark MXFP4 (#41870)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-02 20:47:47 -07:00
Richard WangandRichard Wang 482b062d28 [Feature] serve pplx-decider decision checkpoints on /v1/systemone (#42183)
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
2026-10-03 10:18:53 +08:00
WenhaoZhangandXiaoyu Zhang 41c6ffea89 [diffusion] fix: keep residual_gate_add on the JIT CUDA path for cont… (#42240)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-03 10:12:11 +08:00
Xiaoyu Zhang 391e665dab [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (#41251) 2026-10-03 10:04:48 +08:00
MickandMick Qian 65c7425a56 [diffusion] fix: respect explicit residency over pipeline preload hints (#41622)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-03 09:54:21 +08:00
Liangsheng Yin a8b54f138e [Session] Run streaming sessions only on UnifiedRadixCache; reject unverified tree caches (#42295) 2026-10-02 18:15:59 -07:00
8ff02f5820 [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) (#41729)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
2026-10-02 17:57:42 -07:00
Kevin MiandClaude Opus 5 65f759144d [HiCache] Let the MiniMax-M3 K-only host pool join a host pool group (#42166)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-02 17:00:13 -07:00
Baizhou Zhang 3bde8eb2d8 [CP 1/5] Remove the CP adapter and separate interleave transport from boundary reduction (#42036) 2026-10-02 16:50:20 -07:00
Zhang, Jiejing cbe070b6bf [Docs][AMD] Update GLM-5.2 MI355X daily image to 20260930 (#42278) 2026-10-02 15:39:07 -07:00
b69a5f296a [AMD] Add opt-in MiniMax-M3 TP4 indexer context partitioning (#41488)
Co-authored-by: ThomasNing <thomas.ning@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-10-02 15:34:29 -07:00
fa7784d140 [unified-memory] Copy page envelopes in place during compaction (#42252)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-02 15:12:14 -07:00
e8fab85a02 [KDA] Add an opt-in decode-parity mode to the Triton multi-token recurrence (#42251)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-02 14:59:56 -07:00
19794f5743 [sgl-router] Add SGLang's /v1/classify (#42258)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:56:16 +08:00
Liangsheng Yin 8a328e867b [mem_cache] Rename the tree's request-level insert_req to checkpoint (#42202) 2026-10-02 14:42:35 -07:00
2ec9e3b11d [eplb] Warm default-group NCCL P2P transports before KV-cache sizing (#42026)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: luccafong <luccafong@users.noreply.github.com>
2026-10-02 14:34:55 -07:00
ebf161cea4 [sgl-router] Add SGLang's /v1/rerank (#42205)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:33:40 +08:00
Kan WuandKan Wu 8c9030ce3f [sgl-router] Unify reorg session and cache affinity modes (#41990)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-03 05:21:22 +08:00
Yuhao Wang 0098b9d29d [Spec] Fix Qwen3.5 text model EAGLE3/DFLASH aux-layer capture (#42002) 2026-10-02 14:05:01 -07:00
raghotham a64725d407 [HiCache] Fall back to host memory when no cgroup fs is mounted (#42120) 2026-10-02 13:59:40 -07:00
7dd93b01a5 [PD] Make the prebuilt last-token H2D copy non-blocking under overlap (#42113)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: bbus <bbus@users.noreply.github.com>
2026-10-02 13:48:39 -07:00
Liangsheng Yin 07e1a4f082 [mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public (#42194) 2026-10-02 13:46:32 -07:00
354ed2a7b6 Size the VMM graph-input exchange by the widest input across ranks (#42125)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: oulgen <oulgen@users.noreply.github.com>
2026-10-02 13:45:32 -07:00
4a4dad1cf1 [Metrics] Label scheduler stage wall time by sampled forward overlap (#42025)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-02 13:43:53 -07:00
ChangLiu0709andNikolai Protasov f6fcda8827 [AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)
Co-authored-by: Nikolai Protasov <289648307+amd-nprotaso@users.noreply.github.com>
2026-10-02 11:57:03 -07:00
Shuwen Wang 89f21671bb fix: preserve QSA indexer state through HiCache (#39893) 2026-10-02 23:49:38 +08:00
kkandwunhuang a74f259f38 [AMD][V4.1][*/N] Fuse MXFP8 activation quant into producer kernels on gfx950 (#42055)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-02 08:34:44 -07:00
Alex O. P.andPo-Han Huang e667ab10d5 [diffusion] quantization: ConvRot INT8 online W8A8 for DiTs (sgl-kernel + convrot_int8_customkernel) (#38040)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-02 23:26:28 +08:00
Xiaoyu Zhang 1a2eb6c426 [Kernel] Use Cake softmax for qualified SM103 sampling workloads (#42004) 2026-10-02 23:06:43 +08:00
bb9a820f09 [diffusion] chore: make perf dump one writer per replica with no first-request NCCL setup (#41711)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:44:10 +08:00
Michael 2cd92d10b9 [AMD] Skip the AITER #6042/#5967 patches on gfx1250 (#42219) 2026-10-02 21:10:01 +08:00
edee4308bc [sgl-router] Add OpenAI /v1/embeddings (#42191)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:18:24 +08:00
1330e437e1 [sgl-router] Add native /generate endpoint (#42147)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:09:04 +08:00
Michael e020b389b6 [AMD][CI] Partition stage-b-test-1-gpu-small-amd-mi35x to stop the 30-min timeout (#41973) 2026-10-02 19:29:20 +08:00
MickandMick Qian 0f71ec8656 [diffusion] UX: deduplicate SM120 FP8 fallback warnings across layers (#41542)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-02 19:25:15 +08:00
Michael 544e46c080 [AMD][CI] Fix VLM MMMU nightly max_tokens for CoT prompt (#41979) 2026-10-02 18:59:05 +08:00
YAMYandPo-Han Huang 895525f062 [Mamba] Warm cache COW kernel before serving (#36266)
Signed-off-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-02 18:52:37 +08:00
Alex 1db33bc367 [PD][NIXL] Fix prepared dlists for KV entries of different lengths (DeepSeek-V4) (#42005) 2026-10-02 17:59:31 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 41bd213c5f [Test] Add get_swa_key_page_size to the Q8KV8 sparse-prefill fake KV pool (#42211)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 16:00:15 +08:00
Mohammad Miadh AngkadandMohammad Angkad d954c88a4f [Fix] Update stale source-patch match in dumper comparator e2e test (#42179)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 00:37:37 -07:00
Mohammad Miadh AngkadandMohammad Angkad e7d9c0299b [Docs] Keep the last CUDA 12 image tag fixed during install version bumps (#42201)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 00:17:13 -07:00
sglang-botandsglang-bot d61a9c28ba chore: bump docs install version to 0.5.21 (#42013)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-01 23:16:26 -07:00
Liangsheng Yin 50be533d09 [mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes (#41520) 2026-10-01 21:41:45 -07:00
MingyiandMick Qian b906cd3f45 doc: refresh README (#41967)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-02 12:11:33 +08:00
saatwiknagpal bf8adf9602 [diffusion] fix: fix int32 offset overflow in SubBlock router kernels (#41965) 2026-10-02 11:22:53 +08:00
Khoa Pham 6be84e1b72 [Fix] Select the Marlin MoE runner for MiMo-V2 packed MXFP4 experts on SM90 (#42172) 2026-10-01 20:01:03 -07:00
Yuwei An ef867fa40d [Fix][DSV4.1] SWA page size with bounded replay (#42128) 2026-10-01 18:50:43 -07:00
jiaryangandThomas Wang 58f0d250ec [ROCm] Use the fused MLA absorb + RoPE + KV-write kernel for decode-sized forward modes only (#41533)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-01 18:45:29 -07:00
kkandwunhuang 29f6d408c0 [AMD][V4.1][*/N] Fix shared-expert fusion accuracy and speed up MoE routing on ROCm (#42011)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-01 18:19:06 -07:00
Kedar PotdarandClaude Opus 5.5 8acd36b56b [Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX AgentX recipe (#42109)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:11:29 +08:00
Qiaolin Yu 67eab57057 [qwen 3.8 next] Fuse NEXTN verify and draft graph input preparation (#41175) 2026-10-01 17:17:34 -07:00
caihuali95andCheng Wan 2a3655fea1 [unified-memory] Name the fused KV translate for what it computes (7/7) (#40331)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:14:03 -07:00
df530479c6 [unified-memory] Stride the KV translate kernel and route every translate through it (6/7) (#40330)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:12:14 -07:00
9855e9c6f8 [unified-memory] Remove the kernel-page multiplier plumbing (5/7) (#40329)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:11:25 -07:00
4c4e352a4b [unified-memory] Mark the write loc physical and check it at every write door (4/7) (#40328)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:10:37 -07:00
cdadb1c79a [unified-memory] Token-major dense views for the unified memory pool (3/7) (#38592)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:09:38 -07:00
caihuali95andCheng Wan 5df52e6382 [unified-memory] Build paged KV views through one helper (2/7) (#40327)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:08:42 -07:00
caihuali95andCheng Wan d3dbb71e5a [unified-memory] Derive KV row addresses from strides, not shapes (1/7) (#40326)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:07:47 -07:00
shyeh25andBrayden Zhong 5423a4d885 [MegaMoE] Preserve W13 layout for ModelOpt NVFP4 experts (#39388)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-10-01 16:12:34 -07:00
f17f7705a5 [AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 (#39273)
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: long10024070 <long10024070@users.noreply.github.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: sunxxuns <126995791+sunxxuns@users.noreply.github.com>
Co-authored-by: Long Luong <long.luong2+personal@onemount.com>
2026-10-01 15:59:44 -07:00
Jan Bernlöhrandjbernloehr 80bb3fb651 [Fix] Select the MXFP4 MoE runner for MiMo-V2 packed experts on SM100 (#41668)
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com>
2026-10-01 15:33:30 -07:00
Kan WuandKan Wu a21dcbd320 [sgl-router] Support engines started with --api-key (--worker-api-key) (#41996)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-01 14:54:59 -07:00
Kan WuandKan Wu d7565215ec [sgl-router] Route to a specific DP rank inside multi-rank workers (--dp-aware) (#41974)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-01 14:50:59 -07:00
Martin Hua 52d5faed5b [Bugfix] Llama4 local attention: read page ids from the graph tables in CUDA-graph capture/replay (page_size > 1) (#38187) 2026-10-01 14:42:22 -07:00
bf2d686a82 Add triton autotune on the Mamba2 SSD kernels (#39130)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-01 14:25:37 -07:00
Jiajun LiandClaude Fable 5.1 dcf4e18283 fix(nccl): disable graph buffer registration for pausable graph pools (#40648)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 12:34:23 -07:00
Shangming CaiandClaude Fable 5 a71c7d19e7 [Router] Document peer bootstrap: flags, RBAC, downward API, probe implications (#42049)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-02 02:20:36 +08:00
3f4bfda13c [Bugfix] Fix DeepSeek V4 Pro TP24 vocabulary-padding failure on Hopper (#31801)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:11:18 -07:00
a0fcebb3bb NIXL: Use stride desc API (#37827)
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-01 10:53:00 -07:00
jain-ria fb92ba1127 [Rust] Wire the gRPC server into the frontend lifecycle (#41964)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-10-01 10:44:56 -07:00
JoeandAlex Nails f03a183719 Fix Kimi-K3 MLA output gate dispatch on non-CUDA devices (#36406)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 08:53:31 -07:00
JoeandAlex Nails 3031091c36 [Kimi-K3] Guard optimized paths by platform (#40269)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 08:53:05 -07:00
Артем Савкин 1bbce57d9e [NPU] [Diffusion] Fix NPU multimodal-gen CI (#41528) 2026-10-01 18:02:37 +03:00
41cbe65de0 [sgl-router] Enforce compatible PD version groups (#41613)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 20:24:20 +08:00
3b2ad1c6ae [sgl-router] Fail fast and cancel decode when prefill fails (#41612)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 19:35:56 +08:00
Aurick QiaoandAurick Qiao 1a0250c5b6 [Disagg] Validate state strides before Mooncake transfers (#41607)
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com>
2026-10-01 18:47:41 +08:00
d669efba84 [AMD][V4.1][*/N] Greedy dspark draft/accept under SGLANG_SIMULATE_ACC_LEN (#41981)
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-01 03:35:28 -07:00
46637965ab [sgl-router] Allow load-only routing without a tokenizer (#41611)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 18:30:35 +08:00
3c4e218779 [AMD][V4.1][*/N] OPUS sparse prefill on gfx950 through layout conversion (#42017)
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: thomawan <thomawan@amd.com>
2026-10-01 03:21:25 -07:00
kkandwunhuang f45c004e18 [AMD][V4.1][*/N] Build DSpark draft metadata inside the CUDA graph on ROCm (#42014)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-01 02:17:13 -07:00
黄孝君 7be0e85658 [CI][NPU][Diffusion] Bump ascend consistency GT to the CANN 9.1.0 baseline (#41500) 2026-10-01 17:13:28 +08:00
Xinyi Song 73ba6513f4 [AMD][V4.1][*/N] Switch the fp8 dense GEMMs on gfx950 to aiter's MXFP8 GEMM (#41970) 2026-10-01 02:04:39 -07:00
Xinyi Song 0224fcbd1c [AMD][V4.1][*/N] Route low-ratio indexer and candidate-block top-k through top-k v2 (#41947) 2026-10-01 01:56:31 -07:00
Cheng Wan 5fab1f0f51 [Fix] Let the aiter DCP ASM decode test run without aiter (#42018) 2026-10-01 16:55:15 +08:00
Thomas Wang 69240ba7d6 [AMD][V4.1][*/N] Pick the gfx950 wo_a batched gemm tile by row count (#41971) 2026-10-01 01:52:39 -07:00
Thomas Wang ffb134188b [AMD] Update aiter version for v41 (#42023) 2026-10-01 01:49:45 -07:00
Shunkangz 266d9d1fa7 [Qwen3.8 CP 1/4] Context parallelism for QSA attention and the sparse indexer (#39721) 2026-10-01 01:28:55 -07:00
b51d4a04f0 [DFlash] Keep grouped-conv taps inside each request's block so NaN/Inf cannot leak across requests (#41736)
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 00:18:37 -07:00
Mohammad Miadh AngkadandMohammad Angkad 42f2e4fcb7 [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer (#41987)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-01 00:16:28 -07:00
DarkSharpness 3ed6367d3f [DSV4/DSA] Name the FlashMLA KV format and drop the V4.1 support probe (#41337) 2026-10-01 00:02:18 -07:00
d010e50ad6 [PPU][1/N] CI: Add backend registration and runner preflight (#39788)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 00:01:17 -07:00
amd-danli103 9c3262d855 [AMD][DSV4] feat: enable PD-disagg with fp8 unified_kv on gfx950 (#39166) 2026-09-30 23:42:31 -07:00
315080a0de [sgl-router] Exclude portless prefills and retry bootstrap discovery (#41610)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 14:32:14 +08:00
Shangming CaiandClaude Fable 5 c78fa09290 [Router] Drop the snapshot client's redirect-following fallback (#42010)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:31:53 +08:00
Eric Zhang 947fee9384 [PD] Fix block scale registration for mixed full/SWA KV dtypes (#41924) 2026-10-01 14:25:04 +08:00
Xiaoyu Zhangandkernel-design-agents 77b64e353f [KDA+Kimi K3] Speed up LTX-2.3 QK norm and split RoPE on H200 (#41459)
Co-authored-by: kernel-design-agents <334018530+kernel-design-agents@users.noreply.github.com>
2026-10-01 14:21:31 +08:00
Peng Wu 11500701f1 [DSV4] Reserve the FULL logical page in the c4 indexer pool like its KV pool (#41163) 2026-09-30 23:19:27 -07:00
Shangming CaiandClaude Fable 5 047be72a8c [Router] Harden peer-bootstrap edges: refuse redirects and metric docs (#41999)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:15:42 +08:00
chien-an-chen 99c9d65b32 [AMD] AITER MLA DCP decode ASM path (opt-in) (#40750) 2026-09-30 22:47:55 -07:00
billishyahao dc7defd294 [Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens (#38991) 2026-09-30 22:35:25 -07:00
ashwini rathi 86807b87dd [XPU] Fix test_qsa_indexer_xpu after defer_expansion change (#41915) 2026-09-30 22:34:10 -07:00
Ke Bao 5733bcaad3 Declare HiCache host pools for DSA indexer (#40913) 2026-10-01 13:08:26 +08:00
f02828ea39 fix(dsv4): ship the fp8 unified_kv rope pool through the direct external linker (#40471)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: billishyahao <bill.he@amd.com>
2026-09-30 21:46:56 -07:00
Mohammad Miadh AngkadandMohammad Angkad 40e1bb0f36 [Docs] Add NVIDIA NVFP4 checkpoint to GLM-5.3-Flash cookbook (#41986)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-01 12:15:30 +08:00
3b6d37a56e [Router] Gate readiness on peer bootstrap, and make it observable (13/13) (#40699)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 11:55:46 +08:00
ebdeba2fee [Fix] Emit Responses reasoning content-part events (#41789)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-01 11:15:24 +08:00
Mohammad Miadh AngkadandMohammad Angkad 99c59a2ea6 Drop the GLM-5.3-Flash breakable prefill chunk-size pin (#41837)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 19:26:58 -07:00
Shuwen Wang bb62f125aa [HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free (#41346) 2026-10-01 10:24:51 +08:00
Shuwen Wang c9d6c966d0 [HiSparse] fix: reject speculative decoding and mixed chunk at startup (#41347) 2026-10-01 10:24:15 +08:00
stanleys12andClaude Opus 5 3fb721917d [Fix] Require admin auth for /load_lora_adapter_from_tensors and /set_trace_level (#41322)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-01 10:23:12 +08:00
Shuwen Wang b62b21e7f4 [HiCache][PD] fix: keep decode load-back polling out of rank-local collectives (#41393) 2026-10-01 10:22:30 +08:00
zijieandZijie Chen 0148d9a2fd [AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm (#41497)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
2026-09-30 19:05:49 -07:00
MickandMick Qian 2ab1b537f8 [diffusion] docs: recommend docker for gpu deployments (#41966)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-01 09:52:31 +08:00
36330b8c08 [Router] k8s e2e for cache-aware peer bootstrap (12/13) (#40698)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 09:42:59 +08:00
Shafeeq IbraheemandYanbin Jiang b4df214948 Fix LoRA usage-counter accounting across request lifecycle paths (#31808)
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-09-30 18:37:53 -07:00
Sageandishandhanani baae0195d9 [ci] fix renderer image publish (#41936)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-09-30 18:27:11 -07:00
jain-ria ebccddce19 [Rust] Implement the Rust gRPC adapter for native generation (#41766)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-30 18:17:42 -07:00
Yu-Yun Changandkk 6af651ea0c [AMD][Quark] Serve the Kimi-K3 MXFP4 checkpoint on ROCm (#40811)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
2026-09-30 18:14:15 -07:00
leonardHONGandlianghougong a702970ecb [Perf] Use FA4 by default for MiMo on SM100 (#41886)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
2026-10-01 09:05:12 +08:00
Sage b37e6f79fc [ci] grant ci permissions to sagearc (#41935)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-30 17:21:41 -07:00
Liangsheng Yin 63321e3693 [Scheduler] Move internal-state readback and updates into a collaborator (#41950) 2026-09-30 16:40:57 -07:00
Liangsheng Yin 517cfa218c [Test] Prune redundant scheduler unit tests (#41952) 2026-09-30 16:30:18 -07:00
Jan Bernlöhrandjbernloehr 488869c2d0 [Fix] Keep MiMo-V2 processor available without TorchCodec (#41667)
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com>
2026-09-30 15:28:16 -07:00
826f15bf9c [Fix] Preserve inline instructions for Responses and Kimi K3 Messages (#40077)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-30 15:20:52 -07:00
Sage 8e4bcc5ca9 [rust-renderer] sglang-processor lib (#41896)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-30 15:09:35 -07:00
f050ab194a fix(spec): enable Qwen3.5 EAGLE3 capture and streaming overlap (#41154)
Co-authored-by: Yuhao Wang <59914901+jin-chuan@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <95566987+hnyls2002@users.noreply.github.com>
2026-09-30 14:41:46 -07:00
Cheng Wan a9c97c9f69 [Feature] Add --attn-dp-size and deprecate --enable-dp-attention (#41818) 2026-09-30 13:34:18 -07:00
Jason Mancusoandhnyls2002 201c4b9b2a [RL, Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl (#34201)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-30 13:32:26 -07:00
Cheng Wan 986cb8268b [Refactor] Enter draft TP scopes by attention ownership and drop ModelRunner.tp_group (#41817) 2026-09-30 13:28:50 -07:00
Cheng Wan e7f6a99313 [Refactor] Add make_pp_layers so models stop handing their PP position down (#41816) 2026-09-30 13:28:34 -07:00
Cheng Wan 00659ac099 [Refactor] Stop passing models' layers the placement they already read (#41815) 2026-09-30 13:28:17 -07:00
Cheng Wan c7fa37a5ca [Refactor] Let FusedMoE's weight-loading helpers read the layer's MoE-TP rank (#41814) 2026-09-30 13:27:58 -07:00
Cheng Wan 499e86db98 [Refactor] Keep only the TP and PP groups on the model runner (#41813) 2026-09-30 13:27:31 -07:00
Cheng Wan a09aedf1ca [Refactor] Drop model placement attributes and parameters nothing reads (#41812) 2026-09-30 13:27:11 -07:00
Cheng Wan b0cc246ff8 [Refactor] Drop placement values nothing reads (#41811) 2026-09-30 13:26:51 -07:00
Cheng Wan 4f3ee94d25 [Fix] Make the Solar model constructible and runnable (#41810) 2026-09-30 13:26:32 -07:00
Cheng Wan a659e6461f [Fix] Size the prefill delayer's gather buffer to its TP group (#41809) 2026-09-30 13:26:12 -07:00
Cheng Wan ecc0f0dec2 [Fix] Pass the draft's attention ownership to DFLASH's eager LiLiCorr scope (#41808) 2026-09-30 13:25:54 -07:00
Cheng Wan e289989933 [Fix] Shard MoE WNA16 and Quark INT4-FP8 weights by the MoE placement (#41807) 2026-09-30 13:25:38 -07:00
Cheng Wan 3f20738846 [Fix] State the configured MoE-DP width in the weight-cache fingerprint (#41806) 2026-09-30 13:25:19 -07:00
Cheng Wan e27b6e0971 [Fix] Read TensorCast's WORLD placement under its current names (#41805) 2026-09-30 13:25:03 -07:00
YAMY 4407a21f3c [Spec][PP] Fix hybrid recurrent-state commit and micro-batch pairing under PP x speculative decoding (#40001) 2026-09-30 13:05:39 -07:00
547286bb5a [Deps] Bump transformers to 5.17.0 (#39012)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-30 12:52:02 -07:00
Mohammad Miadh AngkadandMohammad Angkad 036a3b3e14 [Test] Lower the SM120 NVFP4 KV GSM8K threshold to 0.60 (#41799)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 12:50:42 -07:00
Mohammad Miadh AngkadandMohammad Angkad eae808903f [Deps] Bump FlashInfer to 0.7.0.post1 (#40709)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 12:42:19 -07:00
4dfc0ac131 [BugFix] Pass token-major Q/K tensors from Gemma-3 to RadixAttention (#37984)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-09-30 12:05:41 -07:00
Kevin MiandKevin Mi 3a398442bf dsv4.1-amd: serve DeepSeek-V4.1 on gfx950 (#41308)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-30 11:47:14 -07:00
9f9f9e702a [HiCache] Attribute buffer-mode storage hits against the joint device match (#41758)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-09-30 11:46:30 -07:00
zijiexia d26ec4690d docs: remove unreliable DeepWiki badge (#41851) 2026-09-30 10:51:36 -07:00
Shuwen WangandShangming Cai aec4b799e6 [PD][Mamba] fix: free the COW mamba slot of decode requests dropped before preallocation (#41451)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-30 17:50:44 +00:00
jain-ria 21ed9fbf44 [Rust] Extract shared runtime.v1 protobuf bindings (#41726)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-30 10:13:08 -07:00
Sohom chakrabortyandShuwen Wang 28c5e7f5cb [Fix] Stop PD-decode queue_time from counting decode time before a retraction (#41380)
Signed-off-by: Sohom Chakraborty <sohomchakraborty.iitkgp@gmail.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-09-30 23:17:03 +08:00
bd66ce343e [diffusion] nightly: measure every framework with one client end-to-end methodology (#41689)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 19:52:20 +08:00
4b3f63ec70 [Router] Prove a bootstrapped replica answers like the one it copied (11/13) (#40697)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 19:19:26 +08:00
4c930125c7 [Router] Ask the fleet when a graft's splice goes unwitnessed (10/13) (#40696)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 18:59:21 +08:00
Cheng Wan 53225fadf4 [Fix] Stop the namespace census at the leaf a read names (#41838) 2026-09-30 03:27:51 -07:00
f8500474c9 [Router] Share one fleet-wide fetch across a discovery burst (9/13) (#40695)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 18:07:51 +08:00
jacky.cheng 47dcde70f4 [Docs] Add MI355X FP8 agentic recipe to the Qwen3.5 cookbook (#41849) 2026-09-30 17:44:19 +08:00
Yuan Luoandluoyuan.luo 57c81258d4 [KDA] Enable the ptx_kda prefill backend on SM100 (B200 / GB200) (#41445)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-30 17:24:51 +08:00
Yuan Luoandluoyuan.luo c0296186b7 [KDA] Fix ptx_kda prefill NaN without a gate lower bound and workspace growth (#41572)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-30 17:23:54 +08:00
paulzhang-tmandKe Bao 8055ccd2cd [Linear Attention] Expose GDN/KDA prefill hooks and auxiliary cache accounting (#40227)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-09-30 17:19:02 +08:00
EdwardXuyandEdwardXuy 9b7464c8dd ci: temporarily disable A5 (950) nightly job during power maintenance (#41856)
Co-authored-by: EdwardXuy <EdwardXuy@users.noreply.github.com>
2026-09-30 17:17:39 +08:00
jiaryang e9b0d0c0a3 [AMD] Gate flashinfer and TRT-LLM DSA paths on CUDA (#40911) 2026-09-30 02:10:02 -07:00
Michael d7cce54885 [AMD] Resolve QSA packed-varlen decode to aiter on HIP (#41513) 2026-09-30 02:01:15 -07:00
amote-i 964c45cf31 [NPU] [DOC]: add Kimi-K3 NPU PD disaggregation recipes (#41717) 2026-09-30 16:42:46 +08:00
4d4d3f2e5c [Router] Sweep the fleet for a peer snapshot and hand it to the pump (8/13) (#40694)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 16:34:12 +08:00
45c8ddddd3 [diffusion] feat: check the final MP4 in-process instead of spawning ffprobe (#41825)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:26:39 +08:00
WenhaoZhang 0931a72eb2 [diffusion] feat: support MiniMax H3 RL (#34365) 2026-09-30 16:23:56 +08:00
Michael b87a241a6f [AMD] Add Qwen3.8-Flash-Next-FP8 nightly validation (#36901) 2026-09-30 00:54:36 -07:00
682f4ef32f [ROCm] Select DSA indexer top-k wave size by arch (wave32 on gfx1250) (#41687)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-09-30 00:47:04 -07:00
Shuwen Wang 51cae5f303 [HiCache][PD] fix: clamp the decode restore to the prefix promised to prefill (#41450) 2026-09-30 15:26:15 +08:00
heziiopandsglang-npu-bot ec0006673a [NPU] Add decode context parallel support for dsa models (#37787)
Co-authored-by: sglang-npu-bot <sglangnpu@163.com>
2026-09-30 15:22:16 +08:00
Shangming Cai 8c75ad9e62 chore: bump mooncake version to 0.3.13.post1 (#41678) 2026-09-30 15:16:54 +08:00
Zhaoyi Li 3b537e96e2 [AMD][DI][CI] Leave the GLM-5.2 MTP decode room to load its Triton kernels (#41135) 2026-09-30 00:02:42 -07:00
Zhaoyi Li 25de118ae3 [AMD][DI][CI] Give the dsv4pro-fp4 MTP decode more memory headroom (#41238) 2026-09-29 23:57:25 -07:00
Zhaoyi Li 4207f1ae23 [AMD][DI][CI] Record the commit the MI355X nightly tested (#41099) 2026-09-29 23:55:38 -07:00
Zhaoyi Li f85ca901e2 [AMD][DI][CI] Fix four failures in the MI355X disaggregation nightly (#41025) 2026-09-29 23:54:40 -07:00
24ee67d2fa Readme refresh (#41787)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-09-29 23:48:38 -07:00
eb9c9ee99d [Qwen4-Exp] Optional fp8 (e4m3) storage for the compressed QSA indexer cache (#39614)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
2026-09-29 22:11:46 -07:00
Qiaolin Yu 88f95f4b87 [qwen next] Fuse QSA KV preparation and sparse block expansion (#40972) 2026-09-29 21:54:34 -07:00
Qiaolin Yu b660100c00 [qwen 3.8 next] Fuse QSA graph replay metadata across draft steps (#41173) 2026-09-29 21:52:28 -07:00
JohnQinAMDandClaude Opus 5.5 964fd6af92 [ROCm] GLM-5.2: gfx950 four-kernel fused DSA indexer decode path (#38583)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:52:00 -07:00
hhhh1252023 be66c1b460 [NPU][CI] Fix stale multimodal-gen job name in fast-fail health check (#41824) 2026-09-30 12:39:23 +08:00
Cheng Wan d41904f05e [Refactor] Build seven more decoders from stage boundaries (#41779) 2026-09-29 21:16:45 -07:00
Cheng Wan 113e6a1496 [Refactor] Simplify layer boundary internals (#41778) 2026-09-29 21:15:58 -07:00
Wenfei (Charles) Qi 7fb85a9d88 [MLX] Fix startup after logical token capacity change (#41314) 2026-09-29 21:15:56 -07:00
Cheng Wan 9ee065e49a [Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends (#41754) 2026-09-29 21:11:37 -07:00
Cheng Wan faa5064f6e [Refactor] Send producer-written residual streams through to_pp (#41753) 2026-09-29 21:11:02 -07:00
Cheng Wan 843f50f5c1 [Fix] Reduce the Sarvam dense MLP through its row-parallel projection (#41752) 2026-09-29 21:10:17 -07:00
Cheng Wan ea855f0daf [Refactor] Leave decoder stacks through final_norm(skip_empty=True) (#41751) 2026-09-29 21:08:19 -07:00
Cheng Wan 01b5f572b7 [Refactor] Rename layer boundary protocols, contracts and helpers (#41750) 2026-09-29 21:05:48 -07:00
cb3ba2f2b7 [XPU] Add dynamic_expert_bias and track_state support (#39618)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-30 12:05:35 +08:00
Alison Shao 8854857a99 [CI] Bound maintenance API requests to 30 seconds (#41800) 2026-09-29 20:02:02 -07:00
Mohammad Miadh AngkadandMohammad Angkad 7b30ff26ee [CI] Evaluate the maintenance gate with github-script instead of jq (#41625)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-29 18:47:51 -07:00
fc9bdc8a3e [MoE] Keep the moe_align_block_size pad fill inside sorted_token_ids (#41759)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: oulgen <oulgen@users.noreply.github.com>
2026-09-29 17:43:17 -07:00
Baizhou Zhang e5cec303ae [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL (#41777) 2026-09-29 17:28:30 -07:00
sglang-bot cae69be584 chore: bump sgl-kernel version to 0.4.8 (#41784) 2026-09-29 17:19:21 -07:00
DarkSharpness 57d4adeb83 [Kernel] Expose FlashMLA kv_format in sgl-kernel sparse decode (#41336) 2026-09-29 17:16:52 -07:00
Jason Mancuso ba3e04c063 [CI] Grant CI permissions to four contributors (#41775) 2026-09-29 16:42:01 -07:00
Cheng Wan 8d2d874758 [Fix] Release consumed residual contributions (#41749) 2026-09-29 16:38:05 -07:00
Jiajun LiandShi Dong 3d4953839c [Fix] MoE: require TopK layer_id to ensure routed expert captures (#40980)
Co-authored-by: Shi Dong <hello@dongshi.me>
2026-09-29 16:29:27 -07:00
Jiajun Li f0e4001930 [Fix] Allocate SWA KV pools inside the memory-saver region (#40979) 2026-09-29 16:29:02 -07:00
Yuwei An d3f8a8f4f5 [Spec/Prefill Coordination] Dspark support (#41623) 2026-09-29 16:08:01 -07:00
Liangsheng Yin 211b1d9784 [Refactor] Share PD routing fields across request models (#41756) 2026-09-29 15:11:18 -07:00
37a47737c8 [Perf] Tune SM90 GDN recurrent verify launch for small batches (#41486)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-29 15:03:33 -07:00
Mohammad Miadh Angkadandhnyls2002 4885b563c3 [Test] Demote PD test RDMA openability check to a warning (#41681)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 14:52:31 -07:00
84523d6785 [Spec] Reuse K3 auxiliary outputs across decode CUDA graph sizes (#40159)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 14:24:51 -07:00
jain-ria 46beacf13d [grpc rust] Define semantic frontend contract (#41709)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-29 13:40:04 -07:00
James LiuandClaude Opus 5.5 77a81996ce [PD] Preserve bootstrap metadata in native Messages requests (#39711)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:32:54 -07:00
3c4653b6a3 [Metrics] Fix PD latency histogram accounting (#39706)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-29 13:12:09 -07:00
zijiexiaandClaude Sonnet 5.5 f884231f5a [Docs] Add IQuestLab card logo to cookbook landing page (#41746)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 12:47:21 -07:00
51ace48387 [Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds) (#40275)
Co-authored-by: Gilford Ting <gilford@modal.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 12:20:22 -07:00
ishandhanani 9a16e47437 chore: grant jain-ria CI permissions (#41728) 2026-09-29 11:56:51 -07:00
ashwini rathi bf80732b92 Fix pip install: exclude multimodal_gen/.claude symlink from package data (#41703) 2026-09-29 11:35:11 -07:00
8f2a9638f7 [metrics] Fix non-streaming TTFT by flushing the first output through detokenization (#38600)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 11:24:52 -07:00
Colin Z 875dd41e6f [AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm (#28734) 2026-09-29 11:20:10 -07:00
Chunan Zeng affc602f51 [CI] Extend DeepGEMM GB300 validation timeout to three hours (#41720) 2026-09-29 10:57:20 -07:00
ced5e9fe49 [Kimi] Enable GB300 TP4 and GB200/GB300 TP16 SP collectives (#35330)
Co-authored-by: Ajit Mistry <ajmistry@amistry.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-09-29 10:18:11 -07:00
CYJiang fa090f7755 [MUSA] Fix fused MoE GEMV registration and torchada pin (#41444) 2026-09-29 09:51:08 -07:00
ashwini rathi 98fce73d5b [XPU] publish nightly docker image with sgl-kernel-xpu built from main (#41650) 2026-09-29 20:09:53 +08:00
4a4e1ce014 [Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13) (#40693)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 20:07:37 +08:00
8d1643b21e [Router] Vet a peer's snapshot before it may touch the tree (6/13) (#40692)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:58:11 +08:00
8b3a4cad8b [Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13) (#40691)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:06:13 +08:00
7bcfcf5aa0 [Router] Watch EndpointSlices for sibling router replicas (4/13) (#40690)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 18:56:32 +08:00
c5a38d3998 [sgl-router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb (#41268)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:55:58 +08:00
MichaelandClaude 5828cd4927 [AMD] Add GLM-5.3 MI30x and MI35x nightly accuracy tests (#41602)
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-29 02:49:01 -07:00
Rumit DesaiandXiaoyu Zhang f731e82f09 Fix MUSA detection under torch.compile fullgraph (#41065)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-29 17:26:27 +08:00
Carrie ChenandBrayden Zhong 0e586fd12d fuse shared experts with routed experts in MegaMoE's DeepGEMM (#39313)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-09-29 17:25:10 +08:00
c9cb32e310 [AMD] fix kda decode flydsl import (#40546)
Co-authored-by: Bingxu Chen <bingxche@amd.com>
Co-authored-by: Michael <13900043+michaelzhang-ai@users.noreply.github.com>
2026-09-29 02:20:42 -07:00
Liang Gong 34a1234d21 chore: expose agent skills via .agents directories (#40515) 2026-09-29 17:12:28 +08:00
Michael 05817a40c9 [AMD] Fix GLM-5.3 quark MoE MI35x test runner config (#41464) 2026-09-29 16:46:52 +08:00
Jiacong Fang bd78095030 [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B (#41059) 2026-09-29 00:38:58 -07:00
MickandMick Qian dc1bd46802 [diffusion] feat: support bounded exact conditioning cache across native models (#40470)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-29 15:27:22 +08:00
eeecho c7be3e935b [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts (#34528) 2026-09-28 23:08:44 -07:00
Anupa SajikumarandMa Mingfei 2ff52e3ceb [Intel][XPU]Enable chunked prefill scnearios for XPU with UT (#33804)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:28:25 +08:00
9ae43b0eee [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path (#40828)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:16:03 +08:00
Cheng Wan 5781206833 [PD] Give FakeKVReceiver ensure_abort_notified (#41618) 2026-09-28 21:59:17 -07:00
jain-riaandishandhanani 069523f999 [Rust] Extract a transport-neutral frontend core (#39385)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-09-28 21:18:50 -07:00
Cheng Wan 6dad152490 [Refactor] Document layer boundary contracts and integration (#41557) 2026-09-28 20:18:27 -07:00
Cheng Wan 49e7bd179d [Refactor] Group layer boundary unit tests (#41556) 2026-09-28 20:18:08 -07:00
Cheng Wan de9f850b36 [Refactor] Rename the module to layer_boundary (#41555) 2026-09-28 20:17:50 -07:00
Cheng Wan 701213c867 [Refactor] Retire the layer facade and simplify boundary internals (#41554) 2026-09-28 20:17:33 -07:00
Cheng Wan 9b89b96230 [Refactor] Migrate specialized decoder and overlap boundaries (#41553) 2026-09-28 20:17:17 -07:00
Cheng Wan 49b5e76e45 [Refactor] Construct independent decoder stage boundaries (#41552) 2026-09-28 20:17:00 -07:00
Cheng Wan acffb0d20d [Refactor] Select reduction fusion at the consumer (#41551) 2026-09-28 20:16:44 -07:00
Cheng Wan cb1fa3c399 [Refactor] Capture auxiliary states at residual reads (#41550) 2026-09-28 20:16:26 -07:00
Cheng Wan 42f58b4da7 [Refactor] Carry residual state across stage boundaries (#41549) 2026-09-28 20:15:03 -07:00
Cheng Wan 6f72cc3b64 [Refactor] Centralize decoder output access (#41548) 2026-09-28 20:14:04 -07:00
Cheng Wan 0a8ca85565 [Refactor] Group communicator fusion and CP adapters (#41547) 2026-09-28 20:10:50 -07:00
Mohammad Miadh AngkadandMohammad Angkad e49d7fb170 Remove GLM-4.1V-9B-Thinking from encoder DP MMMU test (#41608)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-28 19:45:21 -07:00
Liangsheng Yin d9074f7943 [Test] Fail fast when PD test RDMA devices are not openable by ibverbs (#41600) 2026-09-28 19:34:58 -07:00
James 6e1df69623 [npu]support NPU 910C L2 memcache offload (#41527) 2026-09-29 10:30:01 +08:00
77091cea68 [Radix Cache] Sync Rust TreeCore and make it the default (#39627)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: alphabetc1 <alphabetc1@users.noreply.github.com>
2026-09-29 10:18:57 +08:00
WenhaoZhang 49d26ff0c2 [diffusion] model: support flux 3 action robot policies (#41066) 2026-09-29 10:17:19 +08:00
79cafec013 [Model] Add IQuest Q1 support and MTP draft (#41590)
Co-authored-by: zelong huang <yzhu@ubiquant.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: zelong518 <zelonghuang05@gmail.com>
Co-authored-by: chenwenxiaolive <cocoshirolive@gmail.com>
2026-09-28 17:21:31 -07:00
Stanislav PetrovandStanislav Petrov b95329a74d [Docs] Add GigaChat 3.5 and GigaChat 3.5 Reasoning cookbook pages (#41118)
Co-authored-by: Stanislav Petrov <stapetrov@sberbank.ru>
2026-09-29 08:16:22 +08:00
7889fbab9a [PD] Keep EAGLE DP graph and token metadata consistent (#32196)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-28 17:15:51 -07:00
Shuwen Wang 89e1316eae [mem cache] refactor: remove the index-K continuous getters orphaned by the CP v1 removal (#41469) 2026-09-28 16:41:07 -07:00
Hrithvik AlexandHrithvik Alex f124c5ceec [Model Loader] Stop checkpoint prefetch after iterator completion (#41588)
Co-authored-by: Hrithvik Alex <hrithvik@baseten.co>
2026-09-28 16:27:09 -07:00
9654e5c4bc dsv4.1-amd: fused mHC boundary and all-reduce + mHC post kernels (#41021)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-28 16:15:08 -07:00
Kevin MiandClaude Opus 5.5 096b066fb4 dsv4.1-amd: gfx950 sparse decode attention and sorted top-k (#41020)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 16:14:28 -07:00
Zhang, Jiejing 26d7539799 [Docs][AMD] Update GLM-5.2 MI355X daily image (#41597) 2026-09-28 15:42:54 -07:00
Trevor Morris b07cb9e9e7 [NVIDIA] Update deepgemm, deep-ep, sgl-kernel in CUDA 13.4 image, use cuda base image (#40987) 2026-09-28 14:12:46 -07:00
Nan Jiang 6fa3fe69e2 [PD] Keep the sampling mask of a replayed rebootstrap token (#41235) 2026-09-28 12:02:27 -07:00
Shangming CaiandClaude Fable 5 2a1c477648 [PD] Defer decode KV release on every transfer failure, not only decode-initiated aborts (#41404)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-29 02:19:51 +08:00
YAMY f209e67767 [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism (#39643) 2026-09-28 10:22:40 -07:00
Kurkur 6624999385 [Fix][NPU] fix dp-attn hang when pin_mem is True on NPU (#40446) 2026-09-29 00:41:43 +08:00
triple-mu 2e7e0802f4 [diffusion] Qwen-Image 2.1: fuse Q/K RMSNorm + RoPE + KV packing into one CUDA kernel and project Q/K/V with one packed GEMM (#41339) 2026-09-28 21:45:39 +08:00
DarkSharpnessandClaude Opus 5.5 8151378c1a [DeepSelect] Add page-table transform to top-k and tighten the layout contract (#41364)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 21:43:57 +08:00
DarkSharpnessandClaude Opus 5.5 0318a8d0af [Doc] Add kernel benchmark rule on L2 cache reuse (#41545)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:25:58 +08:00
Shuwen Wang da3eb6db32 [Unified Memory] Fix Inkling conv-checkpoint track ids written to virtual slot numbers (#41144) 2026-09-28 05:19:19 -07:00
Shuwen Wang bddeb61d94 [Unified Memory] fix: preserve FP8 dtype in unified MHA pool (#38133) 2026-09-28 05:19:03 -07:00
MickandMick Qian 090439bf8f [diffusion] docs: consolidate Qwen-Image 2.1 guidance in its cookbook (#41540)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 20:04:32 +08:00
6aacca2d91 [Router] Name a replica's siblings with --kv-peer-selector (3/13) (#40689)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:49:09 +08:00
Raiden MakotoandRaiden-Makoto fa443412c5 [AMD] [GLM5] Fuse shared expert into AITER MoE on gfx950 (#41161)
Signed-off-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
2026-09-28 03:40:09 -07:00
2f88c65289 [Router] Serve the cache-aware tree at /internal/kv_snapshot (2/13) (#40688)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:38:52 +08:00
47ad1928a2 [sgl-router] Scope input_ids forwarding by renderer: all text chats for DeepSeek-V4 (#41226)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-28 18:37:40 +08:00
karverma-amd 85be04978d [AMD][DSV4] moe: enable shared-expert fusion on the grouped-topk path (megamoe) (#40943) 2026-09-28 03:21:57 -07:00
Denji-kk 6caf0ff4ae perf(multimodal): offload CPU feature hashing with bounded admission (#39539) 2026-09-28 17:51:05 +08:00
MickandMick Qian c02eaadeb2 [diffusion] optimization: populate cpu weight stores before host registration to speedup layerwise-offload initialization (#40439)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 17:45:58 +08:00
Peng Wu 6582425854 Let predicate-registered linear-attention models carry the mamba radix-cache leaves (#41165) 2026-09-28 17:07:35 +08:00
df378bca42 [Router] Give the cache-aware tree a snapshot surface (1/13) (#40687)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 16:59:59 +08:00
Peng Wu 546bfa7221 [Kimi-K3] Merge fused_qkvg_proj into the loader-seeded packed_modules_mapping (#41164) 2026-09-28 16:46:11 +08:00
ashwini rathiandPramod Kumar e8eeff2956 [XPU] Bump sglang-kernel-xpu wheel to v0.3.0 (#41220)
Co-authored-by: Pramod Kumar <144990617+pramodkumar-habanalabs@users.noreply.github.com>
2026-09-28 16:15:12 +08:00
Baizhou Zhang 7ee7bef79d docs: add prefill context parallelism guide and design draft (#39354) 2026-09-28 00:26:02 -07:00
Ke Bao 6405cf2881 Fix chat template cache key order (#41517) 2026-09-28 15:17:09 +08:00
flb_ f4de6abee6 feat(npu): Support returning indexer top-k results (#39060) 2026-09-28 14:46:45 +08:00
Rain Jiang 6c4f36e47a Rust server unify datapath for mm and generate requests (#39679) 2026-09-27 23:36:07 -07:00
e75e3b8a98 [Bugfix] fix(hicache): wait for decode offload before retraction (#30899)
Co-authored-by: agent <agent@local>
Co-authored-by: Kevin Flansburg <6134007+kflansburg@users.noreply.github.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-27 23:24:19 -07:00
6be9c78cf4 [Intel GPU] Xpu/weekly simple model enablement 2026 09 21 (#40664)
Co-authored-by: Amrutha M <amrutha.m@intel.com>
Co-authored-by: charlesxu91 <charlesxu.mi@gmail.com>
Co-authored-by: KMS07 <meher.sai.kotthagattu@intel.com>
Co-authored-by: Avi Fenesh <aviarchi1994@gmail.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-28 13:49:18 +08:00
34f090053e [AMD] [Docker] Remove unused LLVM 18 setup from ROCm TileLang build (#41387)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
2026-09-27 22:41:08 -07:00
25de80780a [AMD] Add GLM-5.3-Flash MI35x nightly test (#36903)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 22:40:03 -07:00
Michael 3d2827236d [AMD] Register Triton data movement tests in PR CI (#41137) 2026-09-27 22:38:28 -07:00
Khoa Pham 55cc90b533 [CI] Real-model Kimi-Linear PD parity at page, DCP virtual-page, chunk and cached-prefix boundaries (#41378) 2026-09-27 20:23:07 -07:00
Xinguo Zhu cc012abd21 [CI] fix CI regression on xeon (#41002) 2026-09-28 10:28:01 +08:00
Yihao WangandMick Qian 8b2ca8ecc2 [diffusion] feat: support multiple task types for pipelines (#38762)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 10:00:45 +08:00
Kurkur 6cbdfec453 [Fix][NPU] Fix performance degradation caused by serial execution of two-stage ACL op calls on graph (#40814) 2026-09-28 09:32:16 +08:00
ashwini rathi 0823651b0b [XPU] Disable test_ngram_corpus on XPU and detect XPU tests dynamically in CI filter (#41075) 2026-09-28 09:09:40 +08:00
Cheng Wan 81f27fb3a7 [Refactor] Replace LayerScatterModes with LayerFacts and remove ScatterMode (#41443) 2026-09-27 16:56:11 -07:00
Cheng Wan 06d012eaca [Refactor] Give a layer its CuTe DSL kernels at construction instead of a subclass (#41442) 2026-09-27 16:55:47 -07:00
Cheng Wan 341d5273bd [Refactor] Build the boundary into any stage with one construction (#41441) 2026-09-27 16:55:25 -07:00
Cheng Wan 644014bc98 [Refactor] Declare each stage's residual read and update, and give each stage its own entry (#41440) 2026-09-27 16:55:04 -07:00
Cheng Wan 0e907b755d [Refactor] Split the layer communicator into a package (move only) (#41439) 2026-09-27 16:54:41 -07:00
Cheng Wan 427c9e05c1 [Refactor] Choose every layer's boundaries from declarations and remove the scatter-mode selection (#41438) 2026-09-27 16:54:18 -07:00
Cheng Wan 55d2ee5546 [Refactor] Step-3.5: complete the dense MLP's sum through ffn_exit (#41437) 2026-09-27 16:53:57 -07:00
Cheng Wan 0971450c16 [Fix] LongCat-Flash under attention DP: branch and merge the dense FFNs through the communicators (#41436) 2026-09-27 16:53:36 -07:00
Cheng Wan 4fac02fe5d [Refactor] Choose MoE layers' boundaries from declarations when moe_dp_size equals attn_cp_size (#41435) 2026-09-27 16:53:15 -07:00
Cheng Wan ecb8e9fc8f [Refactor] Falcon-H1: complete the FFN's sum through ffn_exit (#41434) 2026-09-27 16:52:53 -07:00
Cheng Wan 19eb56fa57 [Fix] Falcon-H1: count the Mamba mixer's output once under tensor parallelism (#41433) 2026-09-27 16:52:31 -07:00
Cheng Wan 31329c222c [Fix] Run MoE layers under attention DP and GQA prefill CP on the declared DP × CP gather (#41432) 2026-09-27 16:52:10 -07:00
Cheng Wan f0ecf15c84 [Refactor] Move CuTe DSL-fused layers onto the declared boundaries and FFN-exit kernel entries (#41431) 2026-09-27 16:51:48 -07:00
Cheng Wan 8c43c667cb [Refactor] Nemotron-H: build each layer's boundaries from its stage and the previous one (#41430) 2026-09-27 16:51:27 -07:00
Cheng Wan 6f370c3bcd [Refactor] Build each decoder boundary from the declarations of its two sides (#41429) 2026-09-27 16:51:05 -07:00
Cheng Wan 2e7f0bb28c [Refactor] Publish the LoRA token layout from the batch's FFN input rows (#41428) 2026-09-27 16:50:43 -07:00
Cheng Wan c289656434 [Refactor] Let the MoE declare whether its skipped reduction is one TP all-reduce (#41427) 2026-09-27 16:50:23 -07:00
Cheng Wan deaa7f1463 [Refactor] Run two-batch-overlap layers on the declared boundaries (#41426) 2026-09-27 16:50:03 -07:00
Cheng Wan 6f2d2437b6 [Refactor] Run MHC layers on the shared boundary steps with MHC's residual operations (#41425) 2026-09-27 16:49:42 -07:00
Cheng Wan f5ae989a0e [Refactor] Choose fully-DP dense and DSA / MLA prefill CP layers' steps from declarations (#41424) 2026-09-27 16:49:21 -07:00
Cheng Wan cad9eaf27d [Fix] Gather a dense FFN's input across attention DP and CP in one DP sum (#41423) 2026-09-27 16:48:59 -07:00
Cheng Wan 1c919e401d [Fix] Keep one copy of CP-replicated rows in the DP gather (#41422) 2026-09-27 16:48:39 -07:00
Cheng Wan 7a833ad0a2 [Refactor] Choose a GQA prefill CP extend's steps from declarations (#41421) 2026-09-27 16:48:17 -07:00
Cheng Wan 380a4d0332 [Refactor] Run every batch of a layer from one BoundarySteps (#41420) 2026-09-27 16:47:55 -07:00
Cheng Wan 9f87d28db6 [Refactor] Choose the LayerNorm SP region's and input-scattered batches' steps from declarations (#41419) 2026-09-27 16:47:32 -07:00
Cheng Wan 846181f1c4 [Refactor] Give the fused prepare_mlp kernels an explicit contract (#41418) 2026-09-27 16:47:11 -07:00
Cheng Wan 446d181e23 [Refactor] Choose the boundaries of plain-TP dense layers and MoE layers from declarations (#41417) 2026-09-27 16:46:50 -07:00
Yoni GozlanandYoni Gozlan 583fada954 Port chat_parsing core (#40477)
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
2026-09-27 16:14:24 -07:00
78eee88113 [Spec] Add LiLiCorr: a candidate-lattice reranker for DFlash drafts (#37462)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 15:36:48 -07:00
Shangming CaiandClaude Fable 5 14ab74ef95 [PD] Fan drain abort ACKs out to every decode peer of the room (#41402)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-28 05:16:23 +08:00
Thomas Wang b252aceffe [AMD] Update v4 cookbook for megamoe, fp8 kv attn, BCG (#41458) 2026-09-27 10:53:21 -07:00
e581520c67 [HiCache] Add the page-unified KV load-back JIT kernel (#39726)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-28 01:45:02 +08:00
Xinyuan Tong d10a09ecf3 Add MiniMax arch fallback to auto parser resolution (#40930) 2026-09-28 01:33:27 +08:00
CuzMi b2800461e5 [Diffusion] Reuse bit-exact packed SwiGLU for Ming-Image (#41266) 2026-09-27 23:57:43 +08:00
1ba84a4b59 [bench] Take each request's prompt length from the server (#39889)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-27 23:57:18 +08:00
Yuan Luoandluoyuan.luo c311bc9611 [VLM] Introduce FA4 into ViT for SM100/SM103 (#41344)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-27 23:04:21 +08:00
588049fe2e [diffusion] fix: run an all-valid attention mask on the backend's unmasked kernel (#41309)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 22:54:46 +08:00
Kangrui Du 84622ce9d5 [diffusion] fix: lora-wrapped linears crash qwen-image and minimax-h3 inference (#41272)
Signed-off-by: rockdu <kangrdu@gmail.com>
2026-09-27 22:54:10 +08:00
a0ba1968b1 [Diffusion] Apply latent-ids and packing to caller-provided initial latents (#34418)
Co-authored-by: aimicahchen <aimicahchen@tencent.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
2026-09-27 18:32:20 +08:00
WenhaoZhang f1e62e3a2e [diffusion] feat: add MiniMax-H3 to ComfyUI integrated mode (#35990) 2026-09-27 17:58:54 +08:00
WenhaoZhang ae47bcd4da [diffusion] feat: minimax-h3 spectrum skip-step + fused RMSNorm/AdaLN (#35684) 2026-09-27 17:56:13 +08:00
WenhaoZhang 2a14f65fd7 [diffusion] feat: support in-place lora merge/unmerge under layerwise offload (#36192) 2026-09-27 17:54:40 +08:00
Liangsheng Yin 37556c1b91 [mem_cache] Replace cache_finished_req with insert_req; release_kv_cache frees and unpins (#41281) 2026-09-27 02:53:16 -07:00
Xiaoyu Zhangandkernel-design-agents 0bc074db83 [KDA+Kimi K3] Speed up SANA-Video residual gate add on H200 (#41305)
Co-authored-by: kernel-design-agents <334018530+kernel-design-agents@users.noreply.github.com>
2026-09-27 16:57:12 +08:00
Qiaolin Yu aa7a976807 [qwen 3.8 next] Fuse small CUDA graph input buffer copies (#41166) 2026-09-27 01:27:56 -07:00
ShunkangzandZhangheng df1fc6ab64 [kv-shard 3/4] Enable Control Plane B (#39964)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-27 14:19:08 +08:00
Jianghaiandaimicahchen acc15c193b [Diffusion] Preserve per-sample rollout trajectories across multi-output merge (#34416)
Co-authored-by: aimicahchen <aimicahchen@tencent.com>
2026-09-27 13:08:45 +08:00
Shuwen Wang 38d865489a [DSV4.1][HiCache] fix: wait for the layer transfer before reading low-ratio index-K (#41345) 2026-09-26 22:04:03 -07:00
Liangsheng Yin d27efca353 [mem_cache] Never free the protected prefix on request release (#41312) 2026-09-26 22:00:02 -07:00
b4584ab838 [sgl-router] Take SGLang's render defaults: --default-chat-template-kwargs and thinking/effort envs (#41221)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 11:01:17 +08:00
Kevin Miandx effb752188 dsv4.1-amd: KV cache layouts, FP4 indexer, compressor and router kernels (#41019)
Co-authored-by: x <x>
2026-09-26 19:44:47 -07:00
Ankith Averineni 425a1f8f24 [AMD] Honor an explicit triton moe_runner_backend for mxfp8 on ROCm (#41377) 2026-09-26 17:18:08 -07:00
Lianmin ZhengandLu Fang fc9e1c8d29 [MemCache] Fix LMCache component cursors and per-cache backend selection (#41328)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-09-26 11:24:39 -07:00
c10fa03fc4 Make sliding-window caching and speculative batch padding extensible (#41325)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: jasonjk-park <jasonjk@fb.com>
2026-09-26 11:24:02 -07:00
Thomas Wang c73f7077eb [AMD] Fix jit broken on rocm env (#41356) 2026-09-26 09:59:59 -07:00
Khoa Pham 7bdd8fe6ec [CI] Merge the Kimi-Linear PD DCP4 nightly tests and drop exact-token parity (#41321) 2026-09-26 09:48:00 -07:00
ArielandClaude Fable 5.1 e1e97e8b49 Pin triton_kernels num_warps for MXFP4 MoE below Hopper (6x gpt-oss decode on RTX 4090) (#41292)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 00:43:25 +08:00
Mohammad Miadh AngkadandMohammad Angkad cdbea5dccf [CI] Move DeepSelect into the attention kernel group to fix the namespace test (#41349)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-26 21:39:00 +08:00
Shangming CaiandClaude Fable 5 710a41ba57 [sgl-router] Book the input_ids forwarding outcome only for built bodies (#41342)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-26 19:39:01 +08:00
DarkSharpnessandClaude Opus 5.5 a947b56bc7 [CI] Add __main__ entry to test_deep_select.py (#41341)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 18:12:41 +08:00
Cheng Wan bd37e7f513 [Refactor] Choose a dense layer's boundaries under attention DP from both sides' declarations (#41257) 2026-09-26 02:41:43 -07:00
Cheng Wan 9cd6616c24 [Refactor] Pick the two-batch-overlap split's layout moves once and remove execute (#41256) 2026-09-26 02:41:09 -07:00
Cheng Wan d3fcf8997c [Refactor] Run the LayerNorm SP region's boundary steps in the communicator itself (#41255) 2026-09-26 02:40:34 -07:00
Cheng Wan 35b369ebba [Refactor] Declare at construction the layers whose FFN completes its own reduction (#41254) 2026-09-26 02:39:58 -07:00
Cheng Wan c081d4aacc [Refactor] Drive an FFN exit's flags and its completion from one selection (#41253) 2026-09-26 02:39:25 -07:00
Cheng Wan 8fc3ce48da [Refactor] Choose prepare_attn / prepare_mlp steps and fused kernels at construction (#41252) 2026-09-26 02:38:03 -07:00
7bc988446d [DeepSeek V4.1] Add DeepSelect JIT kernel. (#40556)
Co-authored-by: DarkSharpness <2040703891@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 17:36:48 +08:00
Hexu Zhaoandhexu.zhao a5c8afab48 [Perf] Mamba2 selective_state_update up to 2x faster on B200 via 8x1 launch config for dstate 128 (+7.5% Nemotron-3-Super serving) (#41223)
Co-authored-by: hexu.zhao <zhaohexu2001@gmail.com>
2026-09-25 23:27:01 -10:00
Sundara Raman Ramachandran 3ed56a341b [Score API] Setwise scoring: CausalLM support (batched + --enable-mis) (#41188) 2026-09-25 23:02:44 -07:00
Mohammad Miadh AngkadandMohammad Angkad 4bb645b8e2 [DSA] Fix the pooled-indexer breakable prefill bridge under DP attention (#41311)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-25 22:52:22 -07:00
8772916e06 [DSv4.1] Move the low-ratio index top-k into dsv4/low_ratio_indexer (#41125)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-25 22:22:57 -07:00
7152aabdcb [sgl-router] Track input_ids forwarding outcomes per chat request (#41185)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 13:09:46 +08:00
Xiaoyu Zhang 38ec649048 [Refactor] Restore logical kernel groups and test organization (#41243) 2026-09-26 12:30:44 +08:00
MickandMick Qian e538489e0e [diffusion] fix: copy small files into overlay materialized trees instead of linking them (#41298)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-26 12:22:47 +08:00
WMCandjacky.cheng dfdbff931d [diffusion] fix: fix Qwen-Image 2.1 default RGBA output (#41150)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-26 12:21:24 +08:00
Lianmin ZhengandLu Fang 3ecb6cd729 [MemCache] Unify component eviction cursors and lock receipts (#41276)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-09-25 20:53:35 -07:00
Olga Miroshnichenko 0e2aac500b [ROCm][Perf] aiter: page-level KV view for gfx950 fp8 page-64 asm prefill (#36505) 2026-09-25 19:23:46 -07:00
Liangsheng Yin f35ec18d8b [Test] Remove more unit tests that mirror implementation or never run in CI (#41297) 2026-09-25 19:22:14 -07:00
Shuwen Wang 4273e7de77 [PD] fix: cache resumed decode-radix requests from root instead of an unlocked re-match (#41261) 2026-09-26 10:21:41 +08:00
Mohammad Miadh AngkadandMohammad Angkad 63d320c723 [DSA] Chunk the kpool indexer MQA logits by query rows under a free-memory budget (#40854)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-25 19:13:13 -07:00
MickandMick Qian 2ee82fe158 [diffusion] model: support Anima Base v1.0 (#41011)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-26 09:53:51 +08:00
Liangsheng YinandDarkSharpness 0967a013c2 [DSv4.1] Move the ratio-1/2 index top-k ops into kernels/ops/attention/dsv4 (#41291)
Co-authored-by: DarkSharpness <2040703891@qq.com>
2026-09-25 18:28:04 -07:00
Xiaoyu Zhang 85374532f2 [diffusion] Fuse lossless Klein packed QK RMSNorm and RoPE on Hopper (#40490) 2026-09-26 09:24:21 +08:00
Liangsheng Yin c842d32b0a [CI] Harden /rerun-test dispatch and partitioning (#41285) 2026-09-25 17:39:50 -07:00
50e50e13ab [DCP] Use logical token capacity for PD admission and load reporting (#39731)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-25 17:39:47 -07:00
Liangsheng Yin 1f6ce4b068 [Test] Remove unit tests that only mirror implementation or never run in CI (#41286) 2026-09-25 16:15:02 -07:00
Liangsheng Yin 2afd343934 [misc] Call all_gather_single / reduce_scatter_single to drop torch deprecation warnings (#41284) 2026-09-25 16:14:13 -07:00
Liangsheng YinandBBuf efd9a40bb8 [PD] Share one head-slice helper across mooncake, mori, and nixl (#39660)
Co-authored-by: BBuf <1182563586@qq.com>
2026-09-25 16:13:18 -07:00
Yuwei AnandClaude Opus 5.5 95047d3464 [Fix] Fix cpu CI fail introduced by pr #38652 (#41287)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 15:55:21 -07:00
Liangsheng Yin 516ab77619 [Test] Route all sgl-eval benchmarks through run_sgl_eval and deprecate run_eval (#41280) 2026-09-25 15:48:38 -07:00
Kevin MiandClaude Opus 5.5 27e883a20d dsv4.1-amd: gfx950 MXFP8 matmul kernels and fp8-grid producers (#41018)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 15:45:40 -07:00
Erik WijmansandEthan Su c421d16563 [sglang][lora] Support DP attention in LoRA backends (#36389)
Co-authored-by: Ethan (Yusheng) Su <yushengsu@radixark.ai>
2026-09-25 12:03:47 -10:00
cf5df82680 [KVCache] Support lmcache unified radix cache (#38652)
Signed-off-by: chunxiaozheng <1179548172@qq.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:08:45 -07:00
Ethan (Yusheng) Su d3e3945e41 [LoRA] Size dense row/column-parallel LoRA buffers from the base linear's real shard (#39379) 2026-09-25 14:01:08 -07:00
Liangsheng Yin f88572a57c [Test] Add in-process sgl-eval adapter and move validated GSM8K tests to it (#41216) 2026-09-25 13:59:17 -07:00
Liangsheng Yin f337f01b80 [Test] Remove dead eval modules and point GSM8K/MMLU docs to sgl-eval (#41215) 2026-09-25 12:50:31 -07:00
ac62ddf164 [Rust frontend] Decode input_ids without untagged buffering (#41246)
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Co-authored-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
2026-09-25 12:47:38 -07:00
Nan Jiang 24b6930076 [Sampling] Stream sampling masks as per-request arrays (#40986) 2026-09-25 12:20:25 -07:00
6d35cee7fc Fix multimodal feature offload races (#40621)
Co-authored-by: cctry <cctry@fb.com>
Co-authored-by: Yongji Wu <yongji@meta.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-09-25 12:18:08 -07:00
metamergebotandYonghao Zhuang 671000aee6 [unified-memory] Honor move gates in float relocation and size auto HiCache from host capacity (#41248)
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
2026-09-25 12:17:28 -07:00
31b931edc3 fix(multimodal): return 400 for corrupt image inputs (#28131)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Justin Tong <justintong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-09-25 19:51:33 +02:00
0154f72b48 [ROCm][Bugfix] Keep quantization for mixed Quark Qwen3.5 MTP checkpoints (#39064)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-25 10:31:29 -07:00
Even Zhou 67bb6a58d0 Revert "[NPU] Fuse FIA KV-cache K/V writes into one npu_scatter_pa_kv_cache call" (#41132) 2026-09-25 23:56:22 +08:00
Xiaoyu Zhang 2f5c9ac43d [diffusion] Enable lossless Cosmos3 Super T2I QK fusion on Hopper TP2 (#40486) 2026-09-25 18:01:06 +08:00
Jianghaiandmickqian a749d84303 [Diffusion] Fix AttributeError in grouped forward_batch by installing the residency manager (#34417)
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
2026-09-25 17:27:38 +08:00
ashwini rathi 62122c838a [XPU] Disable test_ngram_corpus on XPU and extend XPU CI path filter (#41224) 2026-09-25 01:54:05 -07:00
Cheng Wan 3450d68d4e [Refactor] Decide an FFN exit's completion once and declare the group it owes (#41200) 2026-09-25 01:50:35 -07:00
Cheng Wan 26a3214cda [Refactor] Take a layer's last-layer fact from its scatter-mode plan (#41199) 2026-09-25 01:49:33 -07:00
Cheng Wan 8f5a636f8d [Refactor] Build prepare_mlp and the layout moves from named steps (#41191) 2026-09-25 01:48:30 -07:00
Cheng Wan b7f6d04a9a [Refactor] Move Step-3.5, GLM5-Next, Dots3, MiniMax-M3 and Qwen3.5 onto ffn_exit (#41198) 2026-09-25 01:47:43 -07:00
Cheng Wan 963e9fb42d [Refactor] Leave the FFN reduction to the next layer under attention DP (#41197) 2026-09-25 01:47:03 -07:00
Cheng Wan 9d7f44bdbb [Refactor] Carry a deferred FFN all-reduce as UnreducedOutput and complete it in the next layer without the fused kernel (#41196) 2026-09-25 01:46:03 -07:00
Cheng Wan 402df23188 [Fix] Stop counting a deferred FFN sum more than once: replicated TP1 shared expert, dense reduce_scatterv (#41195) 2026-09-25 01:45:00 -07:00
Cheng Wan 1409f469be [Fix] Plan NextN / MTP draft layers as one-layer models and fix the Bailing V2 NextN draft (#41194) 2026-09-25 01:43:55 -07:00
Cheng Wan 51c92e8df1 [Fix] Complete the all-reduce when the flashinfer fused norm declines a batch (#41193) 2026-09-25 01:42:16 -07:00
kk 515f5be77e [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4 (#35619) 2026-09-25 00:47:08 -07:00
Shuwen Wang 434c2e3adc [HiCache] fix: Drain pending backups before internal Mamba write-back (#41092) 2026-09-25 15:06:24 +08:00
Shuwen Wang cbe137727e [HiCache] Give trailing sidecar storage transfers a contiguous prefix_keys chain (#40456) 2026-09-25 15:06:05 +08:00
MickandMick Qian 0fb699ac03 [diffusion] model: support Ming-Image Design and Design-Layer (#41067)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-25 14:44:37 +08:00
metamergebotandXingyu Liu 9c8340a2e5 [DeepEP v2] Let a model package supply its per-rank prefill dispatch bound (#41201)
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
2026-09-24 23:39:03 -07:00
Xinyi Song ec070ec8c8 [AMD] Fix int32 offset overflow in Triton DSv4 KV store kernels (#41159) 2026-09-24 23:11:59 -07:00
Ting SUNandXinyuan Tong 3e7e652900 fix(openai): reject request-supplied chat_template by default (#28135)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-09-25 07:48:34 +02:00
Richard Wang 16d1c93b3d [feat] add a system one compatible /v1/systemone route (#41208) 2026-09-25 12:59:11 +08:00
cec70a4708 [Fix] Keep the target's DP sync slot in draft scopes (#41062)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-09-24 21:48:39 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3fc7a669bf [CI] Move GLM-5.2 layer-split test to extra-b-test-8-gpu-b300 (#41207)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-24 21:25:48 -07:00
e0b4d66733 [Perf] Lazy-load built-in model definitions and nixl_ep at startup (#41061)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
2026-09-24 20:29:33 -07:00
7325b38412 [PD] Honor gracefully_exit in disaggregation event loops and keep non-zero-rank launchers alive on SIGTERM (#40793)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Shiyan Deng <dsy842974287@meta.com>
2026-09-24 20:29:07 -07:00
akhilg-nv 37ebcac3c4 Support XQA backend for SpecDec verify (#32269) 2026-09-24 20:14:46 -07:00
MickandMick Qian 6103a8c743 [diffusion] feat: add opt-in SRT prompt enhancement to image and video APIs (#41095)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-25 10:00:04 +08:00
jacky.cheng 9dc4c5d891 [AMD] Restore non-DCP Mamba checkpoint donation to fix agent-mode cache hit at high conc with HiCache (#40907) 2026-09-24 18:08:10 -07:00
Liangsheng Yin 8ca82118e0 [mem_cache] Drop is_insert from cache_finished_req; release rows from release_kv_cache (#40988) 2026-09-24 17:23:26 -07:00
Shuwen Wangandhnyls2002 a2025b8c4d [DSV4] Account for FlashMLA physical KV page padding in memory budgets (#41091)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-24 17:14:08 -07:00
Sundara Raman Ramachandran 2f2f9d12f8 [Score API] Setwise Scoring Support (#38965) 2026-09-24 16:50:58 -07:00
7c5bdb940c [PD] Add a none decode retraction backup and subclass seams in the PD queues (#41103)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <zqx@meta.com>
2026-09-24 16:13:54 -07:00
466e985f4b [HiCache] Batch buffer-only KV backups within each flush (#40960)
Co-authored-by: oss-sync bot (pranjalssh) <pranjalssh@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-24 16:13:02 -07:00
Caio RochaandCaio Rocha e047e50d4e Add 8-node AllReduce/AllGather and MNVLS algorithm support to MSCCL++ (#37442)
Co-authored-by: Caio Rocha <caiorocha@microsft.com>
2026-09-24 15:51:29 -07:00
Yuwei An 3ec86307f2 Fix mixed chunk prefill with DP speculative coordination (#41179) 2026-09-24 15:41:49 -07:00
6bbd689ab4 [Bugfix] Align DeepSeek-V4.1 reasoning effort budgets (#39929)
Co-authored-by: fengtinglei <fengtinglei@bytedance.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-24 15:21:06 -07:00
Liangsheng Yin 0c578d97fe [DSV4] Size compressed pools from one per-ratio table in DSV4PoolConfigurator (#41049) 2026-09-24 15:19:02 -07:00
Qiaolin Yu a1eb691ab5 [Docs] Enable Qwen3.8 Flash Next NVIDIA NVFP4 on B200/B300/GB300 (#41046) 2026-09-24 14:19:00 -07:00
Liangsheng Yin 182f62d2de [Fix] Patch set_dp_buffer_len_from_batch in DP spec prefill coordination test (#41162) 2026-09-24 13:52:11 -07:00
Shuwen Wang 961404b010 [DSV4] Fix TRTLLM uniform FP8 KV memory budgeting (#41090) 2026-09-24 13:25:00 -07:00
36f59982fa MoE: small-batch sorting path with fused mxfp8 quantisation (#36559)
Co-authored-by: mikevin920 <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-24 13:18:18 -07:00
0b53305f48 MiniMax-M3: MXFP8 dense-only block convert + aiter MXFP8 MoE on gfx950 (#36574)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mikevin920 <mikevin920@yahoo.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-24 13:16:59 -07:00
Qiaolin Yu b129504f0e [qwen 3.8 next] Fuse Qwen PLE gate and convolution preparation for target verify (#40041) 2026-09-24 13:13:34 -07:00
Thomas Wang 1446e24d13 [AMD] Add .co for deepseek v4 fp8 decode kernel and add group decode opt (#41120) 2026-09-24 12:21:34 -07:00
Cheng Wan 5bb850d536 [Refactor] Split prepare_attn into a reduction step and per-quant-format residual steps (#41084) 2026-09-24 12:16:54 -07:00
Cheng Wan a80055d3c6 [Fix] Broadcast requests along attention CP before attention TP (#41083) 2026-09-24 12:16:27 -07:00
Cheng Wan 7c9f74c6e1 [Fix] Step-3.5: stop dense layers from summing their output twice under DP attention (#41082) 2026-09-24 12:15:58 -07:00
Cheng Wan 62ae032c46 [Refactor] Share the MoE output all-reduce between models (#41097) 2026-09-24 12:15:33 -07:00
Cheng Wan 5df1667077 [Refactor] Pass each layer stack's output through a communicator exit (#41081) 2026-09-24 12:15:07 -07:00
Cheng WanandMick Qian 07a3735e94 [Fix] Complete the deferred FFN all-reduce before deepstack addition and aux hidden-state capture (#41080)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-24 12:14:39 -07:00
Cheng Wan 928e683f94 [Fix] Complete the deferred FFN all-reduce before a pipeline-parallel send (#41079) 2026-09-24 12:13:57 -07:00
ChangLiu0709 6a14b80141 [AMD] GLM-5.2 MI355X MXFP4: bump image to 20260923 daily (#41109) 2026-09-24 12:02:01 -07:00
Khoa Pham 1b03d31b31 [Test] Run the Qwen3.5 Triton DCP nightly with the radix cache enabled (#41155) 2026-09-24 12:00:09 -07:00
f404db94d4 [DP attention] Publish DP buffer sizes from a ForwardBatch (#40858)
Co-authored-by: hanminglu <hanminglu@fb.com>
Co-authored-by: oss-sync bot (pranjalssh) <pranjalssh@users.noreply.github.com>
2026-09-24 11:34:19 -07:00
Mohammad Miadh AngkadandMohammad Angkad 86b3558b41 [Fix] Skip the DCP target-verify MLA kernel during FlashInfer autotune (#41138)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-24 11:19:28 -07:00
78b382b1ad Support unified memory decode host pools (#39478)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-09-24 11:02:15 -07:00
Trevor Morris 984994e3aa fix: Triton 3.8 compatbility to support DSV4.1-Flash in CUDA 13.4 image (Rubin) (#40805) 2026-09-24 10:25:35 -07:00
Jackey HuaandClaude Fable 5.1 5bb24e399d [Quant] ModelOpt mixed precision: dispatch block-FP8 MoE experts and derive the block size (#38726)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-24 12:30:01 -04:00
Xinyu JiangandZhiyao Jiang 8ec65e89c1 [AMD] ci: move the Miles ROCm 7.2 nightly build to 7.2.4 (#40387)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-09-25 00:09:19 +08:00
Shangming CaiandClaude Fable 5 77173ffb08 [HiCache] Demote SWA KV to host on write_back eviction instead of dropping it (#40712)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-24 23:56:17 +08:00
Ting SUN 752801e4d1 fix(sampling): validate sampling_seed is an int within int64 range (#28960)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-09-24 17:25:13 +02:00
17a7484c06 [chore] surface the cookbook to users who pip install sglang (#40866)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-24 22:56:41 +08:00
Shangming CaiandClaude Opus 5 cd11037009 [PD] Enable deferred decode-side KV release by default (#41023)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 22:35:11 +08:00
黄孝君 4142235c2b [NPU] Update CANN version to 9.1.0 (#40524) 2026-09-24 21:35:31 +08:00
WenhaoZhang ec7eb6bd79 [diffusion] update code owner (#41130) 2026-09-24 21:31:50 +08:00
Xiaoyu Zhang ea5baf4022 [Refactor] Retire the model-specific Kimi K3 kernel namespace (#40922) 2026-09-24 21:25:00 +08:00
paulzhang-tm 28ec6700da [HiCache] Make host reclamation independent of transfer order (#40512) 2026-09-24 21:03:25 +08:00
3177d10ca6 Refactor the Cute-DSL AR fusion to support DeepseekV2 archs (GLM-5.3, etc.) (#39816)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-09-24 06:00:58 -07:00
ronnie_zheng ce06a14444 [Diffusion] migrate the whole _register_configs from registry.py to the model own config file (#40612) 2026-09-24 19:29:49 +08:00
Xiaoyu Zhang c6565c1a6d [Diffusion] Enable lossless SANA-Video eager conv fusions for 12.6% lower latency (#40388) 2026-09-24 19:21:32 +08:00
Alan Kao a69583f70f [AMD] Add tuned dsv4 shape (#40996) 2026-09-24 04:06:40 -07:00
AMD-yanfeiwang 0a59830114 [AMD][DSV4] fp8 unified_kv decode: wave-aware split count past 40 tokens (#40878)
Signed-off-by: Yanfei Wang <yanfwang@crsuse2-m2m-v2-017.us-east2-a.compute.internal>
2026-09-24 03:41:23 -07:00
Xiaoyu Zhang 26c63335e2 [Diffusion] Fuse lossless LingBot World FP32 normalization (#40425) 2026-09-24 18:41:10 +08:00
Zhang, Jiejing d94d784441 [AMD] Tune Triton sparse MLA on gfx950 and make split-K workspaces graph-safe (#39059) 2026-09-24 03:21:53 -07:00
Mohammad Miadh Angkad bb1c98baa2 Fix TBO child batch missing dp_spec_prefill_coordination_applied (#41096) 2026-09-24 02:44:55 -07:00
Xiaoyu Zhang c85df5becc [Diffusion] Fuse lossless Wan VAE post-ops for LongLive 2 I2V (#40405) 2026-09-24 17:40:07 +08:00
Xiaoyu Zhang 261cb826e6 [Diffusion] Fuse LongCat GELU+cat and support Edit-Turbo BCG (#40384) 2026-09-24 17:27:56 +08:00
MickandMick Qian 174a5f37b9 [feature] add per-item candidate token scoring and calibration (#40826)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-24 17:00:13 +08:00
Cheng Xin 41812afe43 [NPU] Fix DSV4 hard-coding kv dtype (#40510) 2026-09-24 16:53:44 +08:00
Yihao Wang 81f2b43ab5 [ci] pr-gate: add generic require-label input and support pull_request_target (#40999) 2026-09-24 01:04:29 -07:00
pllimax d65503fece [NPU] Remove the LLaDA2.0-mini basic-function test case (#40905) 2026-09-24 15:58:48 +08:00
Yuwei An 1417345f5f [Experimental] Preserve speculative decoding during prefill across DP ranks (#40118) 2026-09-24 00:01:12 -07:00
Liangsheng Yin cae4927fbf [DSV4] Budget the ratio-2 pair state pool in DSV4PoolConfigurator (#41048) 2026-09-23 23:53:46 -07:00
Michael a164c6d9f1 [AMD] Register mem-cache unit tests in PR CI (#40812) 2026-09-23 23:35:51 -07:00
YC Yen-Ching TsengandBingxu Chen 43af9fcbb6 [AMD][DI][CI] Move MI355X disagg nightly to ROCm 10 (#41053)
Co-authored-by: Bingxu Chen <Bingxu.Chen@amd.com>
2026-09-24 14:24:39 +08:00
YC Yen-Ching Tseng 9f6fc55332 [AMD][Diffusion] FlyDSL fused norm kernels on wave32 targets (gfx1250) (#37751) 2026-09-23 23:06:55 -07:00
32290dda2c [AMD] Small-M MXFP4 fused-MoE kernel for gfx950 (Qwen) (#40204)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-23 22:45:33 -07:00
Shuwen Wang 90663ccb41 [DSV4] fix: size the C4 state ring by the page it is addressed by (#40337) 2026-09-23 22:05:32 -07:00
cctryandcctry efff836911 [PD] Add decode host receive for custom transfer backends (#40238)
Co-authored-by: cctry <cctry@fb.com>
2026-09-23 21:57:33 -07:00
Theresa Shan 7a9feacf91 [Docs] DeepSeek-V4 MI355X Pro Official PD pairs with DSpark and UMBP (#41024) 2026-09-23 21:48:27 -07:00
fanxingranandThomas Wang 0dc8b29e15 [ROCm][DSA] Enable AITER fused FP8 indexer writer (#40710)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-09-23 21:46:35 -07:00
Nan Jiang c3685dff42 [Sampling] Add selected/support sampling logprob modes (#40932) 2026-09-23 21:38:19 -07:00
Zhaoyi Li 8eedf616f5 [AMD][DI][CI] Say which image the MI355X nightly ran on (#41026) 2026-09-24 12:18:03 +08:00
Jimmy Shong be495a6174 [Fix] Use cached prefix lengths for FlashInfer full-attention ragged prefill (#40796) 2026-09-23 17:23:25 -10:00
jacky.cheng e2f4fedf04 [AMD] Critical fix enabling Qwen3.8 FP8: restore dropped fused shared-expert weights (#40754) 2026-09-23 20:21:33 -07:00
Shijin ZhangandBrayden Zhong 77983865d8 [Moe] Honor swiglu_limit clamped activation in flashinfer_cutlass runner (#39939)
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-09-23 20:10:03 -07:00
Xia WeiwenandCopilot 5c44214d6a [XPU] Qwen3.8-flash-next enablement (#37213)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-24 11:06:30 +08:00
f4d9d2e670 [RL] Add RL weight-update sessions and support updating spec draft runners (#40777)
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Nan Jiang <59716405+nanjiangwill@users.noreply.github.com>
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: JD-ETH <jaedon.guo@gmail.com>
Co-authored-by: Letian Ruan <ruanlt420@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-23 20:04:57 -07:00
Yueming Yuanandzyzshishui f4b90382a2 [RL] Keep DSA cuda-graph state and the graph pool intact across TMS pause/resume (#40804)
Co-authored-by: zyzshishui <zyzshishui@gmail.com>
2026-09-23 19:55:26 -07:00
Liangsheng Yin cc6e8b0f25 [Fix] Derive per-runner hybrid SWA layer ids on ModelLayerInfo instead of mutating ModelConfig (#40983) 2026-09-23 19:52:55 -07:00
jacky.cheng e98b2f7538 [AMD] Speed up Wan2.2 DiT FP8 attention per-tensor quantization (#34695) 2026-09-23 19:49:26 -07:00
jacky.cheng 8b5d77c268 [AMD] Add a Triton packed sparse decode path for QSA on ROCm (#38876) 2026-09-23 18:59:17 -07:00
amote-i 82cdd72588 [NPU] [DOC] Remove --enforce-shared-experts-fusion from npu docs (#41000) 2026-09-24 09:54:29 +08:00
hanwlax 03fcbe147f [NPU] Support batch invariant FIA graphs for deterministic inference (#39607) 2026-09-24 09:53:17 +08:00
Wang, FangYuan 81b81664a6 Revert "[AMD] Fix DeepSeek-V4 accuracy by not passing num_token_non_padded to MoE topk" (#40946) 2026-09-23 18:38:19 -07:00
Liangsheng Yin 621136e094 [mem_cache] Remove unused helpers in mem_cache, storage backends, and metrics (#40963) 2026-09-23 18:16:44 -07:00
Liangsheng Yin 81cb895b85 [Fix] Recover from stale torch extension locks in every cpp_extension loader (#40989) 2026-09-23 17:53:57 -07:00
ec75d3d30f MiniMax-M3: allocate the lightning-indexer K cache in fp8 on gfx95 (#36549)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-23 17:52:02 -07:00
zijiexiaandClaude Opus 5.5 ffac53d779 [Doc] Add H200 recipes to MiMo-V2.6 cookbook (#40969)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 16:49:35 -07:00
Yueming Yuan 542a043f1b [RL] Release the weight-checker snapshot once compare passes (#37284) 2026-09-23 16:06:02 -07:00
James LiuandClaude Opus 5.5 208f6f7501 [Spec] Support DFLASH for Kimi K3 (#40794)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 15:58:39 -07:00
Alison Shao 3f697899a3 ci: reinstall torch/triton left incomplete by a cancelled job (#40819) 2026-09-23 15:55:02 -07:00
Cheng Wan 0010f56a0d [Test] Remove obsolete configuration migration guards (#40976) 2026-09-23 15:33:54 -07:00
6fe4b66f5c [RL] Keep pause_generation and weight updates from deadlocking each other (#40779)
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Zhichenzzz <zczeng@uw.edu>
2026-09-23 15:13:42 -07:00
Cheng Wan 79fec592a8 [Refactor] Read parallel placement in consumers (#40638) 2026-09-23 14:57:25 -07:00
f2a1366584 MiniMax-M3: wave64 histogram-select decode top-k, and raise kMaxNumBlocks for CUDA graphs (#36560)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-23 14:50:13 -07:00
4fa2c9c1de MiniMax-M3: run the sparse prefill main attention through AITER Gluon paged attention (#36546)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Alex Sun <alex.s@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-23 14:49:10 -07:00
Cheng Wan f766397f2e [Refactor] Trim server configuration and runtime context comments (#40971) 2026-09-23 14:47:13 -07:00
Liangsheng Yin c39a1c0663 [mem_cache] Free the rows below the SWA evict floor on all-SWA request release (#40798) 2026-09-23 14:46:28 -07:00
Cheng Wan ab0c31bd94 [Refactor] Move eleven more MoE models to LayerCommunicator.ffn_exit (#40871) 2026-09-23 14:43:41 -07:00
Cheng Wan e1048a215c [Refactor] Let LayerCommunicator own the FFN exit in Qwen3-MoE, DeepSeek-V2 and GLM4-MoE (#40870) 2026-09-23 14:42:58 -07:00
Cheng Wan 7dd9640752 [Refactor] Compare token layouts instead of group sizes when selecting communicator paths (#40869) 2026-09-23 14:41:55 -07:00
Cheng Wan e31cbc01a2 [Fix] Stop deferring the last layer's FFN all-reduce in five models (#40868) 2026-09-23 14:41:14 -07:00
Cheng Wan 701cf7e47e [Refactor] Run Nemotron-H DP attention through the standard layer communicator (#40867) 2026-09-23 14:40:33 -07:00
Cheng Wan 379e8f916b [Fix] Capture complete Nemotron auxiliary hidden states (#40801) 2026-09-23 14:39:50 -07:00
Cheng Wan 9ebe42238b [Fix] Reduce Nemotron MTP attention outputs once (#40800) 2026-09-23 14:39:04 -07:00
Cheng Wan 0e80c73c92 [Fix] Avoid duplicate residual in LongCat MoE shortcut (#40799) 2026-09-23 14:37:39 -07:00
Liangsheng Yin 954458567e [mem_cache] Remove unreachable RadixCache paths in KV canary and HiCache accessors (#40807) 2026-09-23 13:22:58 -07:00
2909f84e3b [JIT] Add an occupancy-preserving L1 carveout preference (#40767)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Oguz Ulgen <oulgen7@gmail.com>
2026-09-23 11:49:34 -07:00
890a9605ce [Metrics] Log forward and forward+idle occupancy over total wall time (#40802)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-23 11:48:31 -07:00
4bb5611dba [Fix] Give the full prefill CUDA graph replay view the captured bucket's input_ids (#40851)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: fei-xx <135589532+fei-xx@users.noreply.github.com>
2026-09-23 11:48:03 -07:00
f2f223e697 Allow attention layers to opt out of the prefill wrapper (#40683)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
2026-09-23 11:39:22 -07:00
Byron Hsu f2eebd5533 [RL] Fix Kimi K3 expert-count lookup for routed-expert capture (#40700) 2026-09-23 11:18:04 -07:00
7fef014d51 feat(kv-hints): add kv hint envelope to request transport (#38891)
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
2026-09-23 10:31:13 -07:00
Even Zhou 5c154c214d Revert " [NPU] Enable piecewise CUDA graph support on NPU" (#40895) 2026-09-23 23:38:14 +08:00
Xiaoyu Zhang c3e985203a [Diffusion] Remove unused standalone benchmarks and deduplicate kernel tests (#40924) 2026-09-23 22:42:03 +08:00
Xiaoyu Zhang 8993f790be [Diffusion] Fuse lossless SenseNova RoPE for 5% faster H200 inference (#40374) 2026-09-23 22:37:22 +08:00
Xiaoyu Zhang 4e60d70ba4 [Diffusion] Fuse Joy Image Edit QKV concatenation and avoid QK copies (#40494) 2026-09-23 22:20:14 +08:00
Xiaoyu Zhang 172b1b4825 [Diffusion] Accelerate Cosmos3 Edge on Hopper with lossless fusions (#40386) 2026-09-23 21:16:58 +08:00
iridiumine 16e353d93e [NPU] Skip fused gmm1+swiglu for swiglu_limit (SiLU-with-clamp) checkpoints (#40438) 2026-09-23 21:02:54 +08:00
iridiumine 3fdd63a562 [NPU] Fuse FIA KV-cache K/V writes into one npu_scatter_pa_kv_cache call (#40445) 2026-09-23 21:02:39 +08:00
Xiaoyu Zhang abef3efb64 [Diffusion] Fuse rounded SwiGLU for quantized MiniMax-H3 MLPs (#40378) 2026-09-23 19:55:48 +08:00
hanwlax 2d25767759 [NPU] Enable piecewise CUDA graph support on NPU (#28417) 2026-09-23 17:30:30 +08:00
inkcherryandDuyi-Wang 58f622ebc4 Reduce decode bootstrap latency with request-owned speculative KV (#38978)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:24:53 -07:00
TianDi101andDuyi-Wang 4cd63da996 [Unified Cache] Dedup replicated MLA/DSA KV in the UMBP direct linker (#38778)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:15:01 -07:00
a89f849158 [ROCm] Fuse the MLA q absorb into the RoPE + KV-write kernel on gfx950 (#38340)
Co-authored-by: JohnQinAMD <zhengsheng1997@gmail.com>
Co-authored-by: RuibinCheung <ruibzhan@amd.com>
2026-09-23 02:07:54 -07:00
siliangchen-amd 401d5aedf3 [AMD] Drop the unreachable vLLM fallback from ROCm FP8 activation quant (#40879) 2026-09-23 01:59:15 -07:00
Yoray Zack de123f38bb [3/N] elastic-ep: Recapture decode CUDA graphs after scale-up (#33723) 2026-09-23 01:51:30 -07:00
aa0b65c7c7 [AMD] Fix DeepSeek-V4 accuracy by not passing num_token_non_padded to MoE topk (#39804)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-09-23 01:35:56 -07:00
RuibinCheungandxiaobochen-amd 48c3854620 [ROCm] feat: enable aiter allreduce fusion for GLM models (#39790)
Co-authored-by: xiaobochen-amd <xiaobo.chen@amd.com>
2026-09-23 01:24:08 -07:00
MatsueYu d4dcce12d4 [npu] decoding procedure optimization on qwen3.5/3.6 (#35958) 2026-09-23 15:23:51 +08:00
ZiruandNiu Ziru 0b0f947265 [CPU] Add fused_sigmod_mul_cpu operators to the Meta Muse Glimmer model. (#39538)
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
2026-09-23 15:12:43 +08:00
Mohammad Miadh AngkadandMohammad Angkad 86cb4a0f97 [CI] Update GLM-5.3-Flash H200/B200 test args (#40862)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-22 23:21:04 -07:00
Polisetty V R K Jyothendra Varma abca3b2e52 [Intel GPU] Add DeepSeek-V2-Lite-Chat-FP8 gsm8k e2e accuracy nightly test on XPU (#39781)
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com>
2026-09-23 14:11:30 +08:00
ANSHUMAN TRIPATHY eb5e8c84be Speculative Decoding with NGRAM support for XPU (#31362) 2026-09-23 14:04:20 +08:00
Zhaoyi Li aa0feca536 [AMD][DI][CI] Use a node-local model cache on the SPUR cluster (#40813) 2026-09-22 23:01:55 -07:00
Shuwen Wang 66ce8c55cc [HiCache] ci: add HiCache and unified radix rerun group (#40831) 2026-09-23 13:56:24 +08:00
a3d11f9d95 [diffusion] feat: support permanent lifetime for layerwise resident layers (#40599)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 13:49:09 +08:00
28be39f72e [deepep_v2] support GLM-5.3-Flash (Glm5NextForConditionalGeneration) (#40466)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 22:32:34 -07:00
jacky.chengandEddie A. Tsai 58988bee2f [AMD] Add diffusion (Wan2.2) extras to gfx1151 Docker image (#40844)
Co-authored-by: Eddie A. Tsai <56776714+vvvdwbvvv@users.noreply.github.com>
2026-09-22 22:18:55 -07:00
Jeff HansonandJeff Hanson 5b8d8b2f88 [Fix] Keep Inkling automatic tool grammar active across the response (#40468)
Co-authored-by: Jeff Hanson <2469458+jshanson7@users.noreply.github.com>
2026-09-23 13:14:36 +08:00
Shuwen Wang ced0a2c98c [Hisparse] fix: account for MiniMax HiSparse full-pool memory (#40743) 2026-09-23 13:13:48 +08:00
Eric Zhang 9d267655e3 Fix NIXL transfer of MXFP8 KV block scales (#40792) 2026-09-23 13:12:04 +08:00
973fb44471 [Diffusion] Support MiniMax-H3 PDD(Parallel Decoding Distillation) inference (#40568)
Co-authored-by: Forrest Lin <forrestl@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-23 12:57:26 +08:00
Shuwen Wang 1d59ce7c90 fix: partition selective CI reruns into matrix jobs (#40760) 2026-09-22 20:37:02 -07:00
zhaozx-cn 525f14040d [NPU]fix ci hicache oom (#40824) 2026-09-23 11:16:50 +08:00
Shuwen Wang 224a247d5b [HiSparse] ci: add cross-directory rerun test group (#40749) 2026-09-23 11:07:03 +08:00
c2bc456b64 [chore] point agents at the cookbook before test configs (#40818)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-23 10:54:15 +08:00
Shuwen Wang 2bc435d32e [HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler (#40312) 2026-09-23 10:52:03 +08:00
Kan WuandClaude Fable 5.1 2e3568567b [sgl-router] Launch reorg routing with existing policy options (#40766)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 10:18:11 +08:00
Richard Wang 40048f6e51 [dLLM] feat: support DiffusionGemma serving (#34061) 2026-09-23 09:59:31 +08:00
Liangsheng Yin 78980a3b0b [misc] Remove deprecated endpoints, env vars and aliases past two releases (#40795) 2026-09-22 18:14:45 -07:00
Alison Shao 0cd8be351d ci: stop Runner Utilization Report from draining the shared API quota (#40791) 2026-09-22 18:01:40 -07:00
Qi SunandXiaoyu Zhang 06008c170d [dsv4.1]Optimize FP4 indexer by skipping invisible tiles (#40431)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-23 07:48:43 +08:00
Liangsheng Yin 4ce23542bf [HiCache] Remove the unused HiRadixCache (#40787) 2026-09-22 16:17:12 -07:00
Mohammad Miadh AngkadandMohammad Angkad 7b977ce5dc [Fix] Decide the MoE padded-row bound from the layer scatter mode (#40672)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-22 16:02:17 -07:00
Liangsheng Yin d34f7b2326 [mem_cache] Clean up SWA/Mamba radix cache leftovers and drop SGLANG_ENABLE_UNIFIED_RADIX_TREE (#40780) 2026-09-22 15:40:57 -07:00
19ee4b566b fix(function_call): buffer complete DeepSeek DSML invokes (#39632)
Co-authored-by: Xinyuan Tong <justintong0323@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-22 15:38:44 -07:00
Jialin Ouyang b77833c504 [MM] Keep scheduler padding in packed token arrays (#40357) 2026-09-22 15:12:07 -07:00
c19dc43cc7 [AMD] [GLM-5.3-Flash Day 0] Load the MXFP4 MTP draft layer (#39779)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: CHANG <jacchang@crsuse2-slog-003.us-east2-a.compute.internal>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-22 15:07:51 -07:00
jain-ria 01275aab6b fix(grpc): expose native response timeout as a server argument (#40644)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-22 14:57:38 -07:00
Liangsheng Yin 9a53f75124 [mem_cache] Remove the experimental C++ radix tree (#40775) 2026-09-22 14:19:45 -07:00
Faradawn Yang d72629e4ba Add GB200/GB300 hardware to Qwen3.5 (#40770)
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
2026-09-22 14:16:43 -07:00
sglang-botandsglang-bot c08ffef709 docs: sync LMSYS SGLang blog cards (#40655)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-09-22 21:14:07 +00:00
Xinyuan TongandZijie Xia 542c817bfa [Docs] Fix benchmark table column overflow in cookbook deployment panel (#40114)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-09-22 14:07:47 -07:00
3afdde5f05 [AMD] [GLM-5.3-Flash Day 0] Enable the k-pool DSA indexer on gfx950 (#39341)
Co-authored-by: Yikai Zhang <ykzhang12@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
2026-09-22 13:14:30 -07:00
077c319914 [AMD] [GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm (#39778)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-22 13:10:39 -07:00
d6cc283da7 [AMD] [GLM-5.3-Flash Day 0] Enable zero-RoPE TileLang DSA on gfx950 (#38547)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 13:08:25 -07:00
Yuwei An 4cbf290fb9 [Fix] Handle chunked paged MQA metadata in DSV4.1 eager forwards (#40637) 2026-09-22 12:59:04 -07:00
720617bb5b [AMD] Reuse KV gather indices across ASM context prefill layers (#39901)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 12:08:05 -07:00
5c18780c31 [Router] Keep e2e workers inside the job's CUDA_VISIBLE_DEVICES allotment (#40686)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 11:32:12 -07:00
Cheng Wan db73f35f4a Fix the Inkling per-expert sync test and collect it in the weekly CPU run (#40725) 2026-09-22 11:22:07 -07:00
Shangming Cai 6f4c2b9b91 [HiCache] Demote internal-node mamba states on write_back eviction (#40680) 2026-09-23 02:18:58 +08:00
Cheng Wan 91c329cc86 Take the model config out of the parallel group build, and finish retiring the parallel getters (#40707) 2026-09-22 10:55:21 -07:00
Pei Li 66454d704b [Feature] Support --tokenizer-worker-num > 1 in the offline Engine API (#40260) 2026-09-22 10:50:01 -07:00
Sage b4701d672d [rust-renderer] decouple renderer sampling from protocols (#40747)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 10:48:28 -07:00
Sage 8ab21c8a94 [ci] publish renderer image (#40639)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 09:48:12 -07:00
Sage a7b96fdad7 [ci] run cpu ci for renderer-only changes (#40636)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 09:47:48 -07:00
YAMYandzhangxiaolei 6fd98c98b9 [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp (#40501)
Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
2026-09-22 08:46:44 -07:00
c79510cc2a [DSv4.1] Score prefill consumer index layers on candidate blocks with DeepGEMM (#40352)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com>
2026-09-22 22:06:06 +08:00
4a1b69abc8 [diffusion] docs: correct the resident-layer help text to match its scope (#40593)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 20:32:59 +08:00
pllimax 244db08d60 [NPU][CI] Constrain evalscope dependency versions (#40133) 2026-09-22 20:15:11 +08:00
Wookje Han bc3a63ebdb [Fix] Missing SWA eviction during decode preallocation (#40309) 2026-09-22 20:11:40 +08:00
Shunkangz ddebc52f23 [kv-shard 3/4] Enable Control plane (#38468) 2026-09-22 19:57:44 +08:00
2524b61f9b [diffusion] feat: allow a component use retain its layerwise resident set (#40592)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 19:56:21 +08:00
Shangming Cai 861b11f087 [PD] Simplify late-abort quiescent ack branch to else (#40711) 2026-09-22 19:08:47 +08:00
amote-i cb8dab06be [NPU] [DOC] Remove duplicated features in npu docs (#40714) 2026-09-22 19:04:17 +08:00
cctry 8ef6d31cb9 [PD] Preserve abort ACKs until in-flight KV transfers drain (#40645) 2026-09-22 18:46:07 +08:00
Emre AlbayrakandXiaoyu Zhang 9d58189c12 [diffusion] attention: add fp8_fa_sm120 FP8 backend for SM120 GPUs (#40175)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-22 17:44:37 +08:00
debbb5cde9 [AMD] Drop the redundant scale zero-fill before AITER per-tensor FP8 quant (#40557)
Co-authored-by: siliangchen-amd <{"message":"Although you appear to have the correct authorization credentials, the `amd-eng-emu` enterprise has an IP allow list enabled, and your IP address is not permitted to access this resource.","documentation_url":"https://docs.github.com/rest/users/users#get-the-authenticated-user","status":"403"}siliangchen-amd@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-22 17:43:31 +08:00
Michael 3f00fb7e2e [AMD] Register unified KV page-zeroing test in PR CI (#40123) 2026-09-22 02:16:31 -07:00
Zhaoyi Li 70a2bb2a30 [AMD][DI] Keep loopback in UCX_NET_DEVICES (#40641) 2026-09-22 02:15:18 -07:00
jacky.chengandchuyeh 790551c382 [AMD] Skip full-vocab softmax in EAGLE topk==1 draft on ROCm (#35872)
Co-authored-by: chuyeh <chuyeh@users.noreply.github.com>
2026-09-22 01:57:37 -07:00
chuyehandjacky.cheng c2f14bfbc4 [AMD] Use exact CU share for gfx950 segment-plan headroom (#39503)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 01:49:59 -07:00
inkcherry 367e3700cf avoid host sync in DSpark prefill slot expansion (#40111) 2026-09-22 01:20:49 -07:00
d00adfd19c [AMD] Fix deferred Kimi-K3 forget gate in fused in-projection (#39525)
Co-authored-by: Tom Jen <tomjen12@localhost.localdomain>
Co-authored-by: HAI <hixiao@gmail.com>
2026-09-22 01:07:25 -07:00
6412ad8c64 [AMD] Pack Qwen3.5 GDN input projections on ROCm (#39902)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 00:52:40 -07:00
Po-Han Huang (NVIDIA) 771c9d782d [PD] Validate Mooncake EFA allocator compatibility (#39973) 2026-09-22 15:12:29 +08:00
Suraj 1815e490de [observability] Fix negative queue_time for retracted requests (#39312) 2026-09-22 15:08:01 +08:00
jshn9515 c948114d58 [DOC] Update quickstart guide to use sglang serve for launching the server (#40647) 2026-09-22 15:00:35 +08:00
5411 changed files with 649671 additions and 143016 deletions
@@ -1,10 +1,14 @@
---
name: add-jit-kernel
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
---
# Tutorial: Adding a New JIT Kernel to SGLang
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -497,6 +501,32 @@ def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) ->
---
## Register the public entry point (required)
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
operator appears in the registry without importing its implementation:
```python
from sglang.kernels.registry import register_kernel
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
register_kernel(
KernelSpec(
op="elementwise.scale",
backend=KernelBackend.JIT,
target="sglang.kernels.ops.elementwise.scale:scale",
capabilities=frozenset({CapabilityRequirement.CUDA}),
)
)
```
Record the devices the implementation actually supports; the CUDA-only scale
example above does not imply that every JIT kernel is CUDA-only. Runtime and
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
Do not import the implementation eagerly in the group initializer. When an
existing `BaseFusedOp` owns the operation, add the implementation there instead
of registering a second conflicting op/backend pair.
## Step 3 (optional): Tune JIT build flags
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
@@ -611,6 +641,8 @@ if __name__ == "__main__":
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
@@ -619,7 +651,7 @@ Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/j
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
- `use_cuda_graph=False` for kernels that can't be captured.
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
@@ -724,7 +756,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
- `python/sglang/kernels/jit/csrc/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
@@ -5,6 +5,10 @@ description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -25,7 +29,7 @@ Add a new operation that scales each element of a tensor by a scalar factor:
In addition, every new kernel must ship with:
- **Tests** (pytest)
- **A benchmark script** (triton.testing)
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
---
@@ -194,6 +198,21 @@ Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
---
## SGLang integration (required for runtime use)
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
actual device/architecture support, and keep `sgl_kernel` imports inside the
implementation path. SGLang runtime and integration tests import that wrapper.
For this example, use `sglang.kernels.ops.elementwise.scale`.
The wheel-level tests below validate its standalone API/build. They do not
replace CI-registered SGLang correctness tests in
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
`write-sglang-test` for their registration and CI budget; do not register tests
under the shipped `python/sglang/` package.
## Step 6: Write tests (required)
Create `python/sglang/kernels/aot/tests/test_scale.py`:
@@ -241,62 +260,46 @@ if __name__ == "__main__":
## Step 7: Add a benchmark (required)
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
```python
import itertools
import torch
import triton
import triton.testing
import sgl_kernel
from sglang.utils import is_in_ci
IS_CI = is_in_ci()
dtypes = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
sizes = [4096] if IS_CI else [2**n for n in range(10, 20)] # 1K … 512K
factors = [2.0]
configs = list(itertools.product(dtypes, sizes))
from sglang.kernels.jit.benchmark import marker
def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
return input * factor
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
sgl_kernel.scale(input, factor, out=out)
@triton.testing.perf_report(
triton.testing.Benchmark(
x_names=["dtype", "size"],
x_vals=configs,
line_arg="provider",
line_vals=["sglang", "torch"],
line_names=["SGL Kernel", "PyTorch"],
styles=[("green", "-"), ("red", "--")],
ylabel="µs (median)",
plot_name="scale-performance",
args={},
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
torch.mul(input, factor, out=out)
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
@marker.benchmark("provider", ["sglang", "torch"])
def benchmark(dtype: torch.dtype, size: int, provider: str):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
return marker.do_bench(
FN_MAP[provider],
# Pass every tensor through input_args (not a closure) so marker rotates
# them across CUDA-graph calls to defeat L2 reuse.
input_args=(input, 2.0, out),
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
memory_output=None,
)
)
def benchmark(dtype, size, provider):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
factor = 2.0
if provider == "sglang":
fn = lambda: sgl_kernel.scale(input, factor, out=out)
else:
fn = lambda: torch_scale(input, factor)
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
fn, quantiles=[0.5, 0.2, 0.8]
)
return 1000 * ms, 1000 * max_ms, 1000 * min_ms
if __name__ == "__main__":
benchmark.run(print_data=True)
benchmark.run()
```
---
@@ -167,7 +167,7 @@ describes what each one does.
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
4. If all matched jobs are completed and count matches `expected_count` → success
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
**Job specs example** (base-b):
```json
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|---------|-------------|---------------|
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
@@ -396,7 +396,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
docs/docs.json docs/cookbook/<category>/intro.mdx
git commit -m "Add <Display-Name> cookbook"
git push -u origin add-<model>-cookbook
gh pr create --title "Add <Display-Name> cookbook" --body "..."
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
```
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
`{{PORT}}` placeholders so users can override through the Env modal.
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
then parallelism (`--tp`, `--attn-dp-size`), then MoE
flags, then tuning knobs, with `--host` / `--port` last. The playground
engine assumes this ordering when inserting overrides (its anchors target
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
| Axis key | Widget | Use when |
|---|---|---|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
knobs/values that are disabled under the live facts, so stale picks never
emit a blocked combination. Interleave prefill-CP + DP-Attention is
deliberately NOT grayed (combined support is planned upstream even though
current releases assert `dp_size == 1` for interleave) — the engine shows
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
a warning hint below the command box instead.
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
playgroundFeatures: {
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
attention: {
knobs: [
{ id: "tp", label: "TP", values: [
@@ -357,8 +357,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -403,8 +402,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
## 1. Quick start
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
a verified recipe below.
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
<Deployment config={config} />
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
can't do coupled changes), and so does a toggle the legacy page itself labels
with operating-point words — e.g. a `dpattention` radio whose options are
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
even when its flags are uncoupled (`--attn-dp-size N` is a pure
flag add). Any other toggle that only adds/removes its own flags becomes a
Playground axis with the flags baked into cells when the legacy default was
ON — EXCEPT parsers:
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
Caveats discovered in the pilot:
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
@@ -136,8 +136,8 @@ than restating.
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
column must be produced by the SAME harness — flag a page that, say, measures one
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
platform's GSM8K with a historical raw completion harness and another's with
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
(the scores aren't comparable). Either standardize on one harness (matching
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
@@ -162,7 +162,7 @@ Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only f
For a rename where the old key must keep working with a warning:
```python
SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
```
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
@@ -11,6 +11,7 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
- A working SGLang installation (`pip install -e .` or equivalent)
- At least one available CUDA GPU
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
## Step-by-step Workflow
@@ -46,13 +47,11 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
### Step 3: Validate accuracy (sanity check)
```bash
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
```
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- This is a quick sanity check, not a rigorous benchmark.
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
### Step 4: Generate the profile
@@ -110,8 +109,8 @@ for i in $(seq 1 120); do
done
# 3. Accuracy check
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
# Expected: Accuracy > 0.8
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
# Expected: score > 80%
# 4. Profile
python3 -m sglang.test.send_one --profile
@@ -0,0 +1,82 @@
---
name: kernel-organization
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
---
# Kernel organization
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
establishes the namespace; subsequent migrations #32148 and #40922 clarify
logical grouping and remove the old `_jit_` filename prefix.
## Choose the owner
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
and integration tests import from that namespace, including its submodules.
The `sgl_kernel` wheel retains its own public API and packaging tests.
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
routing in `moe`, attention index selection in `attention`, sampling in
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
quantization alone belongs in `quantization`.
- Model-specific files/subpackages and tuning data are allowed **inside** a
logical group. Do not add a model bundle or an implementation file directly
under `ops/`. Propose a new logical group only for a distinct responsibility,
and update `ops._GROUPS`, documentation, and tests together.
- Classify a fused operator by its complete contract. Do not split a fused
kernel into separate launches just to separate norm, RoPE, or quantization.
Split unrelated public entry points that happen to share a source file.
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
wrappers call it from their logical group. Keep process groups, communicator
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
`srt/layers/communication/` need not pretend to be generic interfaces.
## Register the public entry point
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
existing `BaseFusedOp` registration when the operation has interchangeable
backends. Group imports must not import GPU implementations or compile kernels.
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
existing backend dispatch and describe actual device/architecture restrictions
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
checks remain part of the entry point's contract. Registering a torch custom op
does not register it in the SGLang kernel inventory. Predicates, private JIT
factories, reference helpers, and runtime classes are not separate kernel APIs.
## Place and preserve tests
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
- Inventory/selector tests: `test/registered/unit/kernels/`.
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
runtime module. A CUDA allocation does not make a runtime test a kernel test.
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
standalone test/benchmark entry points in production operator modules.
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
execution entry points, and CI stages/runners. Apportion existing time estimates
across split files; do not duplicate the original budget for every output file
or silently drop a registration. Follow `write-sglang-test` for CI registration.
## Verify a migration
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
registry references before deleting the old path. Check relative imports and
package re-exports as well as direct imports. Do not leave forwarding shims.
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
behavior. Check wheel/package inclusion as well as source-tree execution.
3. Preserve module singleton state: every runtime caller must import the same
new communication adapter, not a second copy of its buffer registry.
4. Separate relocation commits from semantic changes. Use
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
and CI registration coverage before/after. Only delete an experimental path
after checking its call sites, flags, source, and dedicated tests together.
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
pre-commit, and relevant GPU tests when available. State exactly which GPU
checks ran; import/AST checks do not prove numerical or performance parity.
Do not infer violations solely from a model name inside a group, a missing
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
@@ -46,8 +46,8 @@ in-flight row as shipped.
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
| --- | --- | --- | --- | --- |
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
@@ -70,8 +70,8 @@ in-flight row as shipped.
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
@@ -148,8 +148,8 @@ Stable entries should be folded into the mainline family rows above.
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
candidate_path=(
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
"<br>python/sglang/srt/layers/layernorm.py"
"<br>python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
),
active_keywords=(
"allreduce_fusion",
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
pattern="AITER allreduce fusion",
candidate_path=(
"python/sglang/srt/distributed/communication_op.py"
"<br>python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/layers/layernorm.py"
),
active_keywords=(
@@ -721,7 +721,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
FusionPatternSpec(
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
candidate_path=(
"python/sglang/srt/layers/communicator.py"
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
"forward_mla.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
@@ -0,0 +1,249 @@
---
name: processor-model-parity
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
---
# Processor model parity
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
server, sgl-router, the renderer). A model is supported only when, for the same
request body, the processor produces **the same prompt text and the same prompt
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
the implementation, and SGLang code exists only where the two differ.
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
Multimodal preprocessing is not covered here: `render_request` fails on media
content parts, so hosts send those chats to the engine. The fixture generator
serves every checkpoint text-only, so a media case records that rejection.
## Done means
- **All text should work**: every text chat the engine accepts renders and
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
the processor cannot reproduce goes to the engine; list it in the PR.
- **Chat and completions go through the engine's `/generate`**, with the router
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
engine's OpenAI route is only the `Unsupported` fallback. That needs the
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
against a live engine.
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
to `rust/sglang-processor/` unless they start with `python/`.
## Where a model's code goes
| Piece | File | Needed when |
|---|---|---|
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
| README row | the `render` table in `README.md` | always |
No new test code: `tests/parity.rs` checks every fixture in the directory.
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
renders the model:
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
`PromptFormatter`, replace that branch.
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
variant holding Dynamo's Jinja `PromptFormatter`, picked in
`ChatFormatter::from_tokenizer_config` from the template text (as
`parser/template_detection.py` picks it, where it has a detector), and build the
template context that `_process_messages` and `_apply_jinja_template` build, like
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
rewrites shapes its load-time probes find the template rejects (a later system
turn, for one); SGLang raises on them, so check those before rendering.
## Method
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
Note whether its tokenizer ships a `chat_template`; that can change the spec.
1. **Trace the Python path for the model.** Write down every transformation
between the HTTP body and `prompt_ids`, in order:
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
Reuse `render/reasoning.rs` for this rather than porting it again.
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
replaces the request effort; server `--default-chat-template-kwargs` merge in.
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
It also reads the server's `--tool-call-parser` and whether the tokenizer has
a chat template. If the deployment depends on the parser, set
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
effort profile reads `encoding/encoding_dsv4.py`).
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
message dumping, content flattening, `_handle_last_assistant_message`, system
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
call, and how the prompt is tokenized.
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
rules live there: dropped turns, `</think>` on empty reasoning, argument
formatting, and the errors it raises.
This list is the **divergence inventory**.
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
which often differ from SGLang's. Diff the low-level encoder against Python's
for the rules in step 1.
3. **Write one request case per inventory item** in the fixture (`name` +
`request` only), then generate the expected outputs from Python (below). Cover
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
each failing case, add the smallest SGLang step that fixes it, with a one-line
doc comment naming the Python source it mirrors. Never patch the fixture to
match Rust.
5. **Verify** text offline, then token ids with the checkpoint cached (commands
below). If sgl-router or the renderer uses the formatter, run their suites too.
## Checklist (DeepSeek-V4 case that covers each)
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
reduced to role and content; null content becomes `""`; history tool calls keep
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
- **Content parts**: which part types survive and the join separator. (`parts`)
- **Tools**: all request tools or only those `tool_choice` allows; field order
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
`tool_schema_type_aliases`)
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
- **Pydantic coercion**: typed request fields are coerced before rendering. For
example, `strict: 1` and `defer_loading: "false"` become booleans, and
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
JSON string; other encoders format them their own way. Keep key order
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
truthiness (`1` is on); strings use Python's yes-word set.
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
a separately tokenized prefix. Check whether content is flattened before the
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
assistant turn's tool calls are discarded, so object checks on their arguments
belong after the split. Hosts reach the model through `render_prompt` too, so
check their continuation handling as well.
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
continuation prefix is encoded alone and loses a leading BOS
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
includes pydantic rejections and Python crashes (a 500 is still a rejection).
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
PR that hosts fall back to the engine for it.
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
reach, fix it upstream and mark the case `known_gap` with the reason. The test
reports the case without failing, and fails once it matches so the marker gets
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
## Generating fixtures
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
renders it through the real `_process_messages`, under the case's
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
writes:
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
are kept as written). An `AttributeError` aborts instead: the stub server lacks
something, so extend it;
- the fixture-level `config` (the `config.json` fields the processor reads, plus
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
checkpoint its `chat_template`, so the test renders offline.
```sh
cd rust/sglang-processor
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
```
When a new spec needs more, extend `serving_chat()` in the script (one place):
- **More server state:** set the extra attribute that `__init__` resolves, for
example `_dsv41_default_reasoning_effort`.
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
recorded by the template's text output. Its `render_request` returns the
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
Pin `revision` to a commit hash and regenerate only on purpose.
## Verify
```sh
cd rust
cargo test -p sglang-processor --locked # text, offline
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
```
The token check loads the pinned commit's snapshot from the cache. When the
snapshot is missing, the test prints `token ids not checked` (visible with
`-- --nocapture`), so confirm that line is absent before claiming token parity.
With the snapshot cached, the test also checks that the checkpoint resolves the
fixture's recorded DeepSeek-V4 profile.
## OpenAI layer
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
once its render parity holds. Two more fixtures then cover the rest of Python's
OpenAI layer:
- `tests/fixtures/reasoning_parity/<parser>.json`, from
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
chunked outputs. A parser is served once `src/think/models/` ports it.
- `tests/fixtures/tool_parity/<parser>.json`, from
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
parser is served once `src/tool_call/models/` ports it.
- `tests/fixtures/openai_parity/<model-id>.json`, from
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
output for it, and the response Python builds from that output. Launch the
engine with the model's cookbook flags, including its parsers. Without the
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
fixture does.
## Pitfalls
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
`dynamo-renderer` or `dynamo-tokenizers`.
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
sent, so such requests stay on the engine's route. Fixtures don't show this; a
live engine does.
- Python mutates the request before rendering (`_validate_request` rewrites tool
schemas), so fixture generators drive the server's own entry points.
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
generator pins them and `tests/parity.rs` clears them, so fixtures do not
depend on the machine.
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
such as `task` and message-level `tools`. Parity is defined on `render_request`;
report host-adapter gaps separately rather than bending the model code.
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
beyond the dispatch arm.

Some files were not shown because too many files have changed in this diff Show More