Compare commits

...
1779 Commits
Author SHA1 Message Date
minke.yu b419518774 deploy(benchb): bump new-stack image to g9i-est-td-148936bcb (hicache prefetch call fix) 2026-10-19 02:13:46 +08:00
minke.yu 148936bcb9 fix(hicache): drop stale completed_tokens arg in _check_hybrid_prefetch_result call
Merge artifact: upstream refactored the method to read
operation.completed_tokens internally (8 params), our call site kept
passing completed_tokens (9 args) -> TypeError at runtime on hybrid
prefetch completion, wedging the prefill engine under high L3 hit load.
2026-10-19 01:52:38 +08:00
minke.yu 6ceccac658 deploy: 阶段A/B 新旧基线 compose(4x2卡 standalone 并行 + 双 1P1D PD 并行)
- bencha: old/new × cp2/dp2 standalone,端口 30500-30800,新镜像独立 autotune cache
- benchb: 旧栈(31000/31200/31400+master 50051) 新栈(31100/31300/31500+master 50061),
  NVLink-intra 四件套,新版 dp2 加 --skip-server-warmup(上游 dspark+dp-attention
  warmup 死锁规避,真实流量路径不受影响)
2026-10-19 00:29:07 +08:00
minke.yu 1d2610bbc2 fix(mm): _collect_image_span_requests produce 3-tuple overlapping (idx,item,span_len)
Merged tree convention (PerImageRequestInfo.overlapping annotation and
upstream _assemble_per_image_chunk) is (idx, item, span_len); our span
collector still emitted 4-tuples (idx,item,start,end) from the pre-merge
version, crashing V4.1 warmup assembly: ValueError too many values to unpack.
2026-10-18 23:05:47 +08:00
minke.yu e766b3a0e0 fix(mm): _owner_span_encoder pass data_embedding_func to _move_items_to_device
Upstream refactor added the third param (defer_move check); our local
owner-span encode path kept the old 2-arg call and crashed V4.1 warmup:
TypeError: _move_items_to_device() missing 1 required positional argument
2026-10-18 22:32:28 +08:00
minke.yu e4391748e9 Merge upstream/main (6c3db0f520, 2026-10-18) into g9i-est-td
对齐上游 main 至 2026-10-18。回退基准 tag: 26.10.18。

上游官方版取代的本地 cherry-pick: #40323/#40335(Prefill CP+stream-sync)、
#40177(Decode DP-attn)、#40333->#38603(BCG 官方版)、#40178(MegaMoE 官方版)、
#42194/#41520(disagg prefill 官方重构, 删除本地 checkpoint_disagg_prefill 影子 shim)、
#42420/#42264/#41607/#42113/#41987/#40637/#41658/#41660/#42716 等。

保留的本地独有优化:
- g9i 网关集成 (sgl-model-gateway, query_cache_hit 经 TokenizerControlMixin 存活)
- hicache L3 io_aux 独立车道 + HC_TIMING 打点 (cache_controller.py)
- #38348 fail-soft hybrid prefetch 降级 (unified_radix_cache.py, 适配新
  storage_existence_cache.pool().invalidate_beyond API)
- V4.1 vision CP 放行 (deepseek_v4.py / deepseek_v4_hook.py / eager_runner.py)
- NVLink-intra host 负载修复 (cuda.bindings 指针探测 + STATE_DATA 通道)
- mooncake standalone dummy 补丁 + rank 后缀主机名
- PD debug 仪表 (_dbg_phase/_sched_idled/record_prefill_drain)
- #40429 FlashMLA smem 崩溃修复 (_FLASHMLA_SCHED_MAX_SHARED_BYTES)

#42035 无需再 revert: 上游已默认关 (SGLANG_ENABLE_DISAGG_PREFILL_CONTINUOUS_
INPUT_POLLING=False) 且硬性要求 attn_dp==1 && attn_cp==1 && 非投机。

冲突 34 文件全部解决, srt 下 1038 个变更文件 py_compile 全过。
运行级验证未做, 待 b300 构建镜像 + 冒烟。
2026-10-18 18:13:02 +08:00
Ilia Yastrebovandishandhanani 6c3db0f520 [Perf] NIXL: prepared dlists for hybrid-model state transfer (#44740)
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-18 16:59:18 +08:00
Xinyuan Tong 80fef63786 [Fix] Inkling: frame constrained output in native message tokens (#44716) 2026-10-18 16:47:56 +08:00
Huang Long 889658a5be [Fix] disaggregation: Reuse startup decode graphs across PD role switches (#43760) 2026-10-18 16:16:46 +08:00
+14 d33dd0566a [Feat] srt: Add muse spark runtime, multimodal support and tests (3/5) (#44666)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-18 01:00:24 -07:00
Shangming Cai 80914b7c9b [Refactor] sgl-router: Rename --max-in-flight to --max-inflight-requests (#44950) 2026-10-18 15:33:58 +08:00
Yuwei An 3a6e1fcf77 [Perf] DSv4.1: Bound encoder BCG metadata memory (#44963) 2026-10-18 00:33:35 -07:00
Richard WangandMick Qian de61ab53d4 [Perf] diffusion: Defer full GC collections while importing the runtime (#44606)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 15:31:10 +08:00
Zhaoyi Li f1cf2e1005 [Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index (#44412) 2026-10-18 00:29:22 -07:00
Zhaoyi Li 396e97e5dd [CI] [AMD] ci/slurm: Add a Mooncake KV-transfer test to the MI355X disaggregation nightly (#44409) 2026-10-18 00:28:34 -07:00
Zhaoyi Li 8a4239fbdc [Fix] [AMD] mooncake: Bump Mooncake to v0.3.14-rc1 for cross-node KV transfer (#44404) 2026-10-18 00:22:55 -07:00
Shuwen Wang ba3fc9441d [Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry (#43559) 2026-10-18 15:16:31 +08:00
jacky.chengandMick Qian f96a7326e6 [Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup (#43952)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 00:13:25 -07:00
Shuwen Wang a11fa06be7 [Fix] hicache: Key MLA Mamba state per TP rank in the file backend (#41907) 2026-10-18 15:10:50 +08:00
Shuwen Wang ffc4573c98 [Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend (#43816) 2026-10-18 15:09:58 +08:00
Shuwen Wang e6c2cd47a9 [Fix] hicache: Keep unified memory off the direct page_first_direct copies (#43333) 2026-10-18 15:09:26 +08:00
Liangsheng Yin df966f2345 [Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers (#44952) 2026-10-18 00:07:55 -07:00
Ziang Li 520df8a491 [Perf] weight-sync: Allocate packed receive buffers from metadata (#39264) 2026-10-18 00:04:10 -07:00
Michael 00f6cdcd1b [Fix] [AMD] kda: Reject match_cutedsl_decode on ROCm and skip its parity tests (#44089) 2026-10-18 00:01:40 -07:00
Michael 3626978a13 [Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel (#43991) 2026-10-18 00:01:11 -07:00
Bowen Wang 3671503e44 [Feat] bench: report goodput and SLO attainment in bench_serving (#43967) 2026-10-18 13:31:48 +08:00
62d4261cf4 [Doc] sgl-router: Rewrite README and add architecture guide (#43592)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-18 13:29:08 +08:00
Thomas Wang e5dc10e73b [Doc] [AMD] dsv41: Update amd cookbook 1016 (#44903) 2026-10-17 22:10:42 -07:00
Shangming CaiandShangming Cai 1ff2c1d71e [Fix] sgl-router: Fill the media fields when converting TextGenerate to Generate (#44951)
Co-authored-by: Shangming Cai <shangming.cai@radixark.ai>
2026-10-18 13:05:45 +08:00
Liangsheng Yin d20d010574 [Fix] deepseek-v4.1: Exchange late-layer DP row counts before the forward (#44948) 2026-10-17 21:46:46 -07:00
Liangsheng Yin 6fe630ae14 [CI] deepseek-v4: Set the KV pool in the BCG replay contract test (#44946) 2026-10-17 20:59:03 -07:00
Baizhou Zhang a8b939da43 Revert "[Doc] cookbook: Add Muse Spark Open SWA ratio calculator" (#44947) 2026-10-17 20:55:00 -07:00
140c57d5bf [Perf] diffusion: classify residual_gate_add operands once per call (#44478)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 11:51:52 +08:00
Liangsheng YinandVincent Zhong ff73d63121 [Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph (#44930)
Co-authored-by: Vincent Zhong <207368749+vincentzed@users.noreply.github.com>
2026-10-17 20:51:18 -07:00
Chunan Zeng a0b77db84d [Perf] DFLASH: build the draft num_token_non_padded scalar with torch.full (#44119) 2026-10-17 20:40:16 -07:00
Kaushik Chatterjee 3acfa2efcc [Feat] runtime: Support tokenized multimodal generation (#42833) 2026-10-18 12:35:51 +09:00
Richard Wang 01f3cba70e [Fix] dLLM: Size decode graphs to the default max running requests (#40941) 2026-10-17 20:22:39 -07:00
531b426281 [Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview (#39475) (#39475)
Co-authored-by: Kedi Wu <kediw@nvidia.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-18 11:15:57 +08:00
zijiexia 3b0a154e60 [Doc] cookbook: Add DeepSeek V4.1 Flash 1017 variant (#44937) 2026-10-17 20:04:41 -07:00
zijiexiaandClaude Sonnet 5.5 df24bca4ba [Doc] popular-models: Replace Kimi-K3 with Muse Spark Open (#44693)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 19:54:45 -07:00
14ad64c411 [Perf] custom_op: call register_custom_op ops directly in plain eager execution (#44158)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 10:10:10 +08:00
MickandClaude Opus 5 d748bede96 [Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention (#34385)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-18 10:03:43 +08:00
8d02b69c11 [Perf] diffusion: Replay Triton launches without the runner's per-launch hook metadata (#44471)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:58:54 +08:00
949e1a2080 [Perf] diffusion: Cache the fused linear-GELU verdict per linear (#44643)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:55:29 +08:00
f53bd436c1 [Perf] distributed: cache the eager launch of custom all-reduce v2 per size (#44642)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:54:58 +08:00
minke.yu c8d06424fa gateway metrics: T25b test fix — le buckets are cumulative 2026-10-18 09:53:17 +08:00
minke.yu 97c7eabb33 gateway metrics: T25b fixup — recorder.handle() for 0.17 API 2026-10-18 09:51:58 +08:00
minke.yu e9b436dc2d gateway metrics: explicit histogram buckets for all g9i/router histograms — kill zero-quantile summaries (P2 T25b) 2026-10-18 09:50:23 +08:00
minke.yu fc63ae5392 gateway g9i: T25 test — relax timing-sensitive wait assertion 2026-10-18 08:34:56 +08:00
minke.yu 9df2d40324 gateway g9i: T25 cleanup — drop dead wrapper fns, cfg(test) the test-only ones 2026-10-18 08:31:31 +08:00
minke.yu 6a8a6e399f gateway g9i: T25 fixup — shadow-mode reservoir eligibility is unreachable, not excluded 2026-10-18 08:27:15 +08:00
minke.yu e6c0737541 gateway g9i: T25 fixup — class ttft reads ShadowResponseContext.class 2026-10-18 08:23:53 +08:00
minke.yu 9374e03f13 gateway g9i: hot-channel admission A/B — reservoir + hardlane policies (P2 T25) 2026-10-18 08:21:21 +08:00
Yuwei An b234b5c32e [Refactor] TCPCG: Consolidate attention eager regions and isolate Inkling policy (4/9) (#42288) 2026-10-17 16:06:24 -07:00
Yuwei An e4512a5a17 [Perf] DSv4.1: Reduce BCG metadata memory (#44920) 2026-10-17 16:05:28 -07:00
Liangsheng Yin 563cd60345 [Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm (#44934) 2026-10-17 15:47:41 -07:00
Yi (Vincent) Zhong 05ce0e980f [Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay (#42901) 2026-10-17 14:12:00 -07:00
05c1539098 [Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker (#41343)
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2026-10-17 13:24:08 -07:00
YAMYandClaude Opus 5.5 3b34d9f8b6 [Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools (#44114)
With an FP8 KV pool on SM10x, keep the packed sparse-decode K/V scratch in FP8 instead of dequantizing to BF16 (BF16-q + FP8-KV TRTLLM-gen decode). Tests cover FP8 pool -> FP8 scratch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 11:50:38 -07:00
minke.yu 86883ae89f gateway g9i: deep placeholder rings anchor classification (P2 T24)
v14 replay (c512, 32836 reqs): T23 chunking works (lookup_depth mean 41,
placeholder 6795 vs engine-hit 7183) but Hot never fired — placeholder
anchors kept suffix_start=0, so every shared-prefix request counted its
whole message list as new work -> ColdLarge. Now a placeholder ring at
depth >= MIN_PLACEHOLDER_DEPTH (2) anchors like a real hit: suffix_start
via ring_msg, matched_node filled (record writes the node on every ring),
anchor_tokens stays 0 (unknown, deliberate). depth==1 placeholders stay
misses (shared-greeting guard). ShadowDecision gains
anchor_is_placeholder; the shadow x engine agreement counter's shadow
axis widens to hit/placeholder/miss via ShadowAnchorKind. KNOWN
LIMITATION pinned in a test: a chunk-ring anchor mid-message hides an
unbounded remainder (wrong-Hot vector for >theta mid-message divergence
after a >=2048B shared prefix). No deploy.
2026-10-17 23:57:44 +08:00
minke.yu 43c2844c18 gateway g9i: intra-message 2048B chunk rings for long-prompt prefix hits (P2 T23)
A per-message chain is blind inside one long message (v13: 9034/9034
depth-0 misses on single-message long-prompt traffic). Text pieces
longer than 2048 bytes now roll into bare chunk rings
xxh3(prev || chunk) before the message-final ring, which carries only
the <=2048-byte tail; texts <=2048 hash byte-identically to pre-T23
(pinned by a verbatim legacy-chain equivalence test). Lookup itself is
unchanged, but classify's suffix_start now maps the hit ring back to
its message index via the new ring_msg sideband (the raw n-depth+1
formula overshoots the message arrays once chains outgrow messages).
No deploy.
2026-10-17 22:33:01 +08:00
4eb606b2ae [Feat] diffusion: support opt-in pi0.5 completion-boundary action batching (#44634)
Co-authored-by: Jiahua Huang <35223412+dadiaokua@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 20:49:44 +08:00
b15a78a881 [Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach (#40171)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 20:48:14 +08:00
23f8e6f7ec [Fix] diffusion: keep ZMQ ingress sockets on their configured port (#43732)
Co-authored-by: Sylvan Liu <syliu@cse-ai-16.amd.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 20:47:29 +08:00
+3 d4191a4624 [Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning (#44493)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
Co-authored-by: Mick Qian <mickqian@radixark.ai>
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com>
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Fabio Camaiora <fcamaiora@nvidia.com>
Co-authored-by: Linda-Stadter <lstadter@nvidia.com>
2026-10-17 20:46:13 +08:00
9771e5a4f0 [Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol (#44662)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:45:07 +08:00
MickandMick Qian e01dfd3cbc [Refactor] multimodal: Replace padding strategies with helpers (#44845)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 20:43:02 +08:00
Shuwen Wang 86e67f7d93 [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation (#43387) 2026-10-17 20:20:49 +08:00
SuperSong 025655068c [Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction (#44097) 2026-10-17 04:56:41 -07:00
SuperSong ee30dc4d40 [Fix] unified-memory: Bound joint reclaim and preserve cached prefixes (#39294)
Bound joint FULL/SWA/Mamba reclaim so feasible allocations recover capacity without evicting unnecessary cached prefixes, and impossible demands preserve the cache. Use exact unsharded paged-extend demand and update allocation regression fixtures.
2026-10-17 04:46:48 -07:00
Shuwen Wang ae28306a74 [Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay (#44220) 2026-10-17 19:13:57 +08:00
Shuwen WangandZhangheng 5fcb07da34 [Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend (#43817)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-17 11:08:17 +00:00
Woody-HuandMick 465ddf9758 [Feat] diffusion/VSA-H3: support sm80 (A100) with a raw-pointer Triton load path (#43823)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-17 18:47:19 +08:00
400c1dfe6e [Perf] distributed: skip the DLPack re-wrap of custom all-reduce results (#44251)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 18:44:31 +08:00
pllimax eb5739b30e [CI] pr-test-npu: Temporarily disable recommend-tests-from-coverage job (#44881) 2026-10-17 18:21:17 +08:00
Khoa PhamandClaude Fable 5.1 a126e8d640 [Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints (#44838)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-17 02:45:53 -07:00
MickandMick Qian 80f40522cd [UX] diffusion: log LoRA merge decisions only on first evaluation (#44376)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-17 16:47:07 +08:00
56a20d7e34 [Perf] kernels: Enter a JIT kernel's device guard only when the device differs (#44644)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:43:27 +08:00
f75232cb42 [Fix] hicache: Order write-back host copies after the forward that produced the KV (#44800)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-17 01:31:40 -07:00
0d13efb2b8 [Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend (#44767)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-17 01:30:50 -07:00
a2953d7b7f [Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay (#44850)
Co-authored-by: Pedram Ghazi <pedram.ghazi@amd.com>
Co-authored-by: David Orman <ormandj@corenode.com>
2026-10-17 01:25:33 -07:00
McZyWu 7a8804edb5 [Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP (#40516) 2026-10-17 16:19:44 +08:00
Liangsheng Yin ef26f6ad49 [Fix] deepseek-v4.1: Keep the cached KV of folded encoder replay rows (#44857) 2026-10-17 00:56:18 -07:00
Khoa PhamandClaude Fable 5.1 bce7b27aa6 [Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8 (#44842)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-17 00:19:35 -07:00
01dc08e23f [Refactor] dsv41: Shard Engram tables on hash-column boundaries (#42787)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-17 00:15:49 -07:00
minke.yu 9e288079e4 gateway g9i: fix non-stream CL truncation on shadow strip + pre-strip breakdown extraction + shadowxengine agreement metric (T22)
- Bug A (user-visible): non-stream passthrough forwarded the upstream
  Content-Length after the shadow strip had shortened the body ->
  cache-hit non-stream responses truncated (~60B, IncompleteRead).
  non_stream_passthrough_response now pins CL to the actual body length.
- Bug B (order): strip_and_extract returned the post-strip JSON to
  shadow_record, so extract_cached_breakdown always missed and
  smg_g9i_shadow_cache_hit_tokens_total never fired. The breakdown is now
  extracted pre-strip and threaded through record_response_json.
- T22 observability: extract_cached_total (standard
  usage.prompt_tokens_details.cached_tokens first, tier breakdown as
  fallback) + smg_g9i_shadow_engine_agreement_total{shadow,engine} 2x2
  counter pairing the request-time lookup hit with the engine's actual
  cached tokens; ShadowResponseContext carries shadow_hit.
2026-10-17 15:11:58 +08:00
Khoa PhamandClaude Fable 5.1 14c4dda24e [Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size (#44837)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-16 23:47:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 9c7b338a19 [CI] jit-kernel: Honor continue_on_error on scheduled runs (#44645)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 22:29:01 -07:00
Mohammad Miadh AngkadandMohammad Angkad 2d8a05e719 [Test] attention: Fix replicated attention test missing full_attention_interval (#44602)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 22:28:37 -07:00
Liangsheng Yin 9259553f09 [Fix] megamoe: Support EPLB rebalance on SM100 (#44847) 2026-10-16 22:22:40 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3fc08b939a [Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache (#44849)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-17 13:22:34 +08:00
Mohammad Miadh AngkadandMohammad Angkad 4fc86f73c2 [CI] glm5.2: Keep EOS in HiSparse needle test output (#44597)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-17 12:20:51 +08:00
Liangsheng Yin 4cbea9d3c7 [Doc] cookbook: Allow PD with DSpark for DeepSeek-V4.1 (#44848) 2026-10-16 21:17:32 -07:00
Xinyuan Tong 86cc2e02ec [Fix] gateway: Drop unused npyz dev-dependency to unbreak builds after zip yank (#44720) 2026-10-16 20:56:31 -07:00
Ziang Li 29a7e19de1 [Refactor] nvfp4_online: Use explicit FlashInfer 4over6 config (#41745) 2026-10-16 20:56:10 -07:00
feixiangpengandzijiexia 9590dc80cc [Doc] hicache: Explain HiCache host pool sizing with a storage backend (#44790)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-17 11:28:37 +08:00
Shuwen Wang 6a39ddf1f2 [Fix] unified_cache: Resize test allocator before allocating backed pages (#44843) 2026-10-16 20:21:49 -07:00
3603824541 [Perf] DSv4.1: Enable decoder bounded replay by default (#44403)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-16 19:52:02 -07:00
minke.yu 738d5a50d0 gateway g9i: shadow semantic gap tests (image-in-prefix/theta boundary/deep hit/shared-prefix repoint/try_consume/reasoning_content) 2026-10-17 10:15:38 +08:00
McZyWu 2d3dd3168d [Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback (#40131) 2026-10-17 10:12:41 +08:00
Khoa Pham ff979e0816 [Fix] megamoe: Respect request-pool limits in decode validation (#44816) 2026-10-16 18:56:38 -07:00
5cbe2371a5 [Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend (#44764)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 18:14:50 -07:00
Shuwen Wang 33b5d26fa6 [CI] lora: Use the canonical Llama model ID in eviction tests (#44771) 2026-10-16 18:13:37 -07:00
b62a48ee15 [Feat] sglang-server: Serve /flush_cache from the Rust frontend (#44766)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 17:24:01 -07:00
d8dd49d21d [Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker (#44785)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-16 16:55:54 -07:00
d230f2c35b [Feat] profiler: Add the forward pass id to detailed step span labels (#44775)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-16 16:51:20 -07:00
36cf028346 [Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id (#44592)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-16 16:47:50 -07:00
Khoa PhamandClaude Fable 5.1 aaf5f397a5 [Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention (#43813)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-16 15:38:33 -07:00
Yuwei An 8e3f11b219 [Perf] DSv4.1: Skip discarded KV rows during window commit (#44783) 2026-10-16 15:05:45 -07:00
a46fe83c70 [CI] tests: Fork the gloo exchange test's ranks from a preloaded forkserver (#44679)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
2026-10-16 14:53:21 -07:00
AaravandAarav d5d87899f9 [Fix] DSA: skip-topk pipeline handoff for owed outputs (#43832)
Co-authored-by: Aarav <211792885+AaravMehta-07@users.noreply.github.com>
2026-10-16 14:35:23 -07:00
Khoa PhamandTrevor Morris 71b60fa46c [Fix] engram: Avoid blocking history copies during PD admission (#44787)
Co-authored-by: Trevor Morris <tmorris@nvidia.com>
2026-10-16 14:29:05 -07:00
2948662026 [Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo (#44622)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-16 13:53:41 -07:00
metamergebotandmetamergebot f39c2693aa [CI] rust-server: Raise the ChunkEvent frame-size cap to 152 bytes (#44773)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-16 13:47:52 -07:00
Chunan Zeng 07950be745 [Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8 (#42811) 2026-10-16 13:43:18 -07:00
Mohammad Miadh AngkadandMohammad Angkad 21791e7023 [CI] [NVIDIA] DSv4.1-Flash: Split GB300 accuracy test into one file per config (#44696)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 13:40:03 -07:00
metamergebotandmetamergebot a31a8c2bbd [CI] tests: Let test_hicache_storage size its host cache to the runner (#44729)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-16 13:38:39 -07:00
Liangsheng Yin a18a908066 [Test] mem_cache: Free SWA peers below the eviction floor in the SWA repair test (#44777) 2026-10-16 12:41:39 -07:00
+14 456231e251 [Feat] kernels: Add muse spark operators and collective GEMM (2/5) (#44665)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-16 10:49:35 -07:00
ormandjandhnyls2002 80c54e6ac5 [Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA (#42716)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-16 10:45:16 -07:00
553068af25 [Fix] rust-server: Honour cache_salt and require_reasoning on /generate (#44533)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 10:38:09 -07:00
453b251dbd [Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll (#44614)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-16 10:37:11 -07:00
metamergebotandjiayisuse 989ccf4863 [Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer (#44605)
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-16 10:36:46 -07:00
63a9b7a713 [Feat] http_server: Make the multi-worker ASGI app path configurable (#44583)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-16 10:36:35 -07:00
26a2f18c02 [Feat] sglang-server: serve /v1/loads from the Rust frontend (#44532)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-16 10:35:37 -07:00
minke.yu e80c8608dc gateway g9i: --g9i-window-fixed pins the admission window, skipping all AIMD dynamics (P2 T21) 2026-10-17 01:24:23 +08:00
Xinyuan Tong a038a0dca0 [Fix] dsv41: Preserve tool-result image order and current-turn thinking (#44751) 2026-10-16 23:39:50 +08:00
Duyi-Wang ed2ebad342 [Doc] [AMD] cookbook: Refresh DeepSeek V4 AgentX PD recipe optimizations (#44454) 2026-10-16 23:34:01 +08:00
Jimmy ShongandJimmy Shong 2c2986be2b docs: add Beam 1.0 deployment cookbook (#44694)
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
2026-10-16 13:29:14 +00:00
minke.yu 5351d5491b gateway g9i: AIMD warmup tuning (init 128, add 64, CLI flags) + prompt_too_long 400 front validation (P2 T20 A+B) 2026-10-16 20:07:49 +08:00
Benjamin TruongandClaude Opus 5.5 7c99ace127 [Fix] dsa: Fix ragged seed indices before MTP draft decode (#42565)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:37:41 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 5e240a200d [Doc] cookbook: Refresh SemiAnalysis AgentX launches for Qwen3.5 and DeepSeek-V4.1-Flash GB300 (#44712)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:31:08 +08:00
Xinyi SongandHAI 4afae5568b [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance (#44657)
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-16 03:11:02 -07:00
2adc76ad61 [Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8) (#41903)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:55:25 -07:00
YC Yen-Ching Tseng 9246acd462 [CI] [AMD] aiter: pin aiter commit to 2bb549 for rocm10.1 gfx1250 (#44620) 2026-10-16 17:50:20 +08:00
kkandDuyi-Wang baf451fcd6 [Perf] [AMD] dsv4: carry aiter DSv4 GEMM tuned rows and mHC tile_m=16 until aiter ships them (7/8) (#44008)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:47:41 -07:00
f05da43d44 [Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) (#30929)
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:39:27 -07:00
42eb556e66 [Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8) (#44007)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-16 02:28:31 -07:00
caihuali95 8d229d96e0 [Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding (#42930) 2026-10-16 02:24:06 -07:00
kkandDuyi-Wang 6efc96abf9 [Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8) (#44006)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:16:06 -07:00
kkandDuyi-Wang 9fde8bcc46 [Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) (#44004)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 02:09:32 -07:00
kkandDuyi-Wang f0ff18353d [Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8) (#44003)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 01:58:00 -07:00
MingyiandZijie Xia 56a3c4fa16 [Doc] cookbook: Add RTX Spark Qwen3.6 NVFP4 recipe (#44707)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-10-16 01:55:41 -07:00
Liangsheng Yin 81c9d804b0 [CI] pr-check-comment: Comment on PRs when lint or title check fails (#44708) 2026-10-16 01:54:38 -07:00
kkandDuyi-Wang ab5858ac8a [Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) (#44016)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-10-16 01:50:16 -07:00
Xinyuan Tongandzijiexia 9baace8456 [Doc] cookbook: refresh MiniCPM-V 4.7 DSPARK benchmarks at the final model PR head (#44310)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-16 01:48:25 -07:00
3d61bdd2c9 [Feat] sgl-router: Serve the remaining native gRPC inference RPCs (#42681)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-16 16:23:17 +08:00
kkandwunhuang 536cdb4b04 [Feat] [AMD] docker: Install the sgl-amd-mono monokernel plugin in the ROCm image (#44621)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-16 01:20:40 -07:00
Liangsheng Yin fad7d95a7f [Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree (#44673) 2026-10-16 01:18:58 -07:00
Baizhou Zhang 06b81b120e [CI] deepgemm: Increase B200 release test timeout to 180 minutes (#44706) 2026-10-16 01:14:54 -07:00
kkandwunhuang bdc4ec15ce [Perf] [AMD] dspark: No per-step host sync in decode (1/4) (#44405)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-16 01:13:37 -07:00
ZiruandNiu Ziru 5dd37af412 [Doc] [XPU] Cookbook: Update the XPU Docker version to 0.5.22 (#44647)
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
2026-10-16 16:13:24 +08:00
minke.yu 8a2e1403a3 gateway g9i: blind-state timeout no longer blocks recovery when pipeline < 0.5W (P2 T19 P1) 2026-10-16 15:59:14 +08:00
minke.yu 1910eccb8e gateway g9i: ledger reconcile clamp on sustained ledger>>engine divergence (subtract-only, 60s fuse) (P2 T19 P0-2) 2026-10-16 15:56:56 +08:00
jiaqiang-dot-liu a448fdf316 [Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode (#44267) 2026-10-16 00:51:55 -07:00
minke.yu 6217b9de46 gateway g9i: TTL reaper for wedged D-ledger bookings (pipeline 120s / streaming 600s) (P2 T19 P0-1) 2026-10-16 15:51:40 +08:00
amote-i 9d394dfbd7 [Doc] [NPU] docs: refresh associated dependency versions in npu docs (#44695) 2026-10-16 15:48:37 +08:00
Ziang Li 7098ec387c [Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE (#44036) 2026-10-16 00:38:50 -07:00
Liangsheng YinandJingshu Chen 3a5d46e946 [Perf] pd: Stage decode-side KV preallocation lengths through pinned memory (#38955)
Co-authored-by: Jingshu Chen <96519051+elmo2019@users.noreply.github.com>
2026-10-16 00:34:56 -07:00
Xiaoyu Zhang 51faf52305 [Fix] monokernels: Honor resolved CUDA graph configuration (#44630) 2026-10-16 15:06:14 +08:00
Baizhou Zhang dfc89580f1 [Doc] cookbook: Add Muse Spark Open SWA ratio calculator (#44687) 2026-10-16 00:00:12 -07:00
amote-i 92580581da [Doc] [NPU] docs: Freeze qwen3-235b-a22b recommended image to a fixed version (#44650) 2026-10-16 14:14:05 +08:00
Mohammad Miadh AngkadandMohammad Angkad 44be30d216 [Chore] deps: Bump FlashInfer to 0.7.2 (#44591)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-15 22:51:21 -07:00
Shu Wang 83940471b4 [Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques (#44387) 2026-10-15 22:50:33 -07:00
+14 e1d20574da [Chore] docker: Add muse spark CUDA image and DeepGEMM patches (1/5) (#44664)
Co-authored-by: 1am9trash <1am9trash@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ethan Che <eche@meta.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai>
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com>
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster>
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: ethche <ethche@gmail.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
2026-10-15 22:46:22 -07:00
minke.yu 9dd446b5c1 gateway g9i: AIMD blind-state recovery threshold prealloc < max(1, max_running/4) (P2 T18 hole-4)
The blind-state (no TTFT samples — non-streaming traffic) slow-add
condition required D prealloc_queue == 0, which is unreachable in
production (prealloc sits at 180+ under load), so the window stayed
pinned at WINDOW_MIN=16 forever — the admission window never recovered
after a cut.

New rule: slow-add when prealloc_queue < max(1, max_running_total/4)
(pipeline < 0.8W condition kept; hold-timeout still forces hold). A
missing snapshot for the node is treated as busy (no blind recovery on
zero information).

Test: aimd_blind_holds_while_prealloc_busy replaced by
aimd_blind_recovery_prealloc_threshold (0/10 recover at limit 64,
boundary == limit holds, production-shape 180 holds, max(1, mr/4) floor
at mr=8, missing snapshot holds).
2026-10-16 13:42:48 +08:00
minke.yu f76eaf0f92 gateway g9i: gates judge on last-known snapshots; blind fallback round-robins instead of cache_aware pass-through (P2 T18 hole-3)
The stale-exemption + blind-pass-through combination fed the 2026-10-16
p3 pileup: the most overloaded node answered /v1/loads slowest, its
snapshot went stale first, and staleness EXEMPTED it from every cap
judgment; with all snapshots stale, regular_plan returned
PrefillPlan::Default and the underlying cache_aware radix policy pinned
every same-prefix session onto that one node (p3: 477 queued, TTFT
~300s) while the drained node (p2) got zero traffic.

a) v1_over_cap / p_real_queue_over / least_wait / sorted_by_wait now use
   last-known values: any snapshot that ever fetched OK participates at
   any age; only a NEVER-answered node is unknown (unjudged, unpicked).
   Stale judgments are counted: smg_g9i_stale_snapshot_judged_total{gate}.
b) All-blind (no node ever answered = true cold start) no longer returns
   PrefillPlan::Default — regular_plan round-robins the pool via a new
   rr_cursor (decision path regular_blind_round_robin). Partial blind
   picks the known least-wait node as before. Prefix affinity lives in
   the shadow table (shadow_route), not the underlying policy.
c) reqs_cap=10 / token_cap=16M / hold 1s/512 / HOT-first / stateless
   per-request exclusion semantics unchanged; check_decode_pool freshness
   (TTL) semantics intentionally untouched.

Tests: v1_over_cap_snapshot_semantics + shadow_path_real_queue_gate_filters
updated to the last-known contract; new t18_least_wait_uses_last_known,
t18_all_blind_round_robin_not_default, t18_partial_blind_least_wait_override.
2026-10-16 13:41:09 +08:00
minke.yu 45fdc97082 gateway g9i: concurrent /v1/loads fetch + detached bootstrap scrape loop + last-known snapshot survival (P2 T18 hole-2)
poll_once fetched 4x /v1/loads + 3x P /metrics SERIALLY, each with a 3s
timeout — one round cost up to ~21s against a 5s snapshot TTL, so one
slow node at peak aged the whole pool's snapshots and blinded every
gate (no_fresh_snapshot on ~30% of decisions, 2026-10-16).

- /v1/loads fetches now run concurrently (JoinSet); per-node 3s timeout
  unchanged, a round costs the slowest node, not the sum. Poll interval
  1s / TTL 5s unchanged.
- The per-P /metrics kv_transfer_bootstrap_ms scrape moves to its own
  10s slow loop (concurrent scrapes); it only feeds observability gauges
  and must never delay the snapshot round.
- A failed fetch no longer overwrites the snapshot with a zeroed
  default: the last-known load survives (merge_poll_result) and keeps
  aging via its original fetched_at; a node that never answered stays
  truly unknown instead of fabricating '0 queue'.
- New gauges: smg_g9i_poll_duration_seconds (histogram),
  smg_g9i_snapshot_age_seconds{worker}.

Tests: t18_poll_failure_preserves_last_known (merge semantics),
t18_poll_fetches_concurrently (two hanging nodes cost one timeout, stay
unknown but remain routing candidates).
2026-10-16 13:33:07 +08:00
Yuwei An 06934575b9 [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage (#44131) 2026-10-15 22:32:55 -07:00
minke.yu f690861265 gateway g9i: book D pipeline ledger at decision time for deferred (miss) admissions (P2 T18 hole-1)
Root cause of the 2026-10-16 window-gate-blindness incident: for a
prefix-table MISS the D pipeline booking happened in shadow_admit at
response-headers time — and in PD mode headers arrive at ~first-token
time (the dispatch join waits on the prefill answer), so the booking
entered the ledger and was promoted to streaming within milliseconds.
The pipeline gauge pinned at 0 while 585 admitted requests awaited
their first token; the d_window_full gate (pipeline >= W) never fired.

Fix: AdmissionMark gains p_booked + a new_deferred() constructor; the
miss branch of shadow_classify_and_admit books the min-T_D D node into
the pipeline ledger at decision time (P side stays deferred — no node
known). shadow_admit consumes the mark by admitting P on the actual
landing node and rekeying the D booking; shadow_release_unconsumed
releases D (and P only when it was booked). Promote/release paths are
unchanged and conserve: admit = pipeline + streaming, complete/abort
= release exactly once.

Also adds the reconciliation gauge smg_g9i_d_engine_inflight{worker}
(running + prealloc + transfer queue, engine truth) beside the ledger
gauges, so ledger-vs-engine divergence is directly visible.

Tests: t11_miss_defers_admit_to_response rewritten as
t11_miss_books_d_pipeline_at_decision_time; new
t18_deferred_promote_release_conservation (promote idempotent,
mid-stream disconnect release, post-consume fallback no-op) and
t18_deferred_abort_pre_first_token_releases_pipeline (double-release
guard).
2026-10-16 13:28:08 +08:00
2a08e261ec [Feat] MiniCPM-V-4.7: add model support (#41993)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-16 13:24:48 +08:00
a2981ab8ea [Feat] sgl-router: Serve native gRPC ChatComplete (#42674)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:11:36 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 fd4b75c696 [Doc] cookbook: Add SemiAnalysis AgentX launches to Muse Spark Open, refresh DeepSeek-V4.1-Flash (#44654)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:45:07 +08:00
Kolade FajimiandXiaoyu Zhang 5117bfbaeb [Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100 (#44213)
Signed-off-by: kolade fajimi <fajimikolade23@gmail.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-16 12:36:02 +08:00
HuangJi 6d70787398 [Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output (#44275) 2026-10-16 12:35:04 +08:00
MickandMick Qian e5274752a8 [Doc] diffusion: align PR title conventions (#44646)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-16 11:29:20 +08:00
minke.yu e4bfdc949b gateway g9i: AIMD blind-state recovery on real engine signals + real-queue P gate + nonstream e2e metric (P2 T17) 2026-10-16 11:16:32 +08:00
Mick 3420baea8c [diffusion] CI: batch performance failure-policy runner scenarios (#44508) 2026-10-16 11:04:05 +08:00
Mohammad Miadh AngkadandMohammad Angkad 6231a617d3 [Test] diffusion: Load NVFP4 block scales through the weight loader in tests (#44595)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-16 10:56:27 +08:00
ashwini rathiandClaude Opus 5.5 75e25a21bd [Fix] [XPU] nightly: Fix rerank, zimage, flux2 and EAGLE parity (#44100)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:27:51 +08:00
Chetan Kumar VermaandClaude Sonnet 5 75e77fca25 [Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions (#43768)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-16 10:22:09 +08:00
Shuwen Wang f9df33494a [Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held (#44386) 2026-10-15 19:17:51 -07:00
Chetan Kumar VermaandClaude Opus 5.5 1e33d2c4bd [Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all (#39989)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 10:08:50 +08:00
4e85e8ed37 [Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it (#44430)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: yeqcharlotte <yeqcharlotte@users.noreply.github.com>
2026-10-15 18:52:33 -07:00
shodocoandmetamergebot 2957ff3165 [Fix] sglang-server: Keep the matched stop token in output_ids (#42637)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-15 18:51:47 -07:00
minke.yu 59e3aee6d9 gateway g9i: pipeline-depth window (streaming excluded) + tiered hold queue + 1s hold budget (P2 T16) 2026-10-16 09:43:54 +08:00
sglang-botandsglang-bot c15edf314d [Doc] blogs: Sync LMSYS SGLang blog cards (#44616)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-16 00:52:14 +00:00
e3961eb6fa [Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler (#44440)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com>
2026-10-15 17:50:43 -07:00
f110b2f801 [Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info (#44531)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 17:50:27 -07:00
d393298736 [Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams (#44537)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 17:49:49 -07:00
metamergebotandYe Qi bda6ef7e01 [Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing (#44568)
Co-authored-by: Ye (Charlotte) Qi <7269575+yeqcharlotte@users.noreply.github.com>
2026-10-15 17:48:50 -07:00
ed8c3139f1 [Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends (#41303)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-15 17:32:50 -07:00
minke.yu 406081d3e3 gateway g9i: admission hold queue (429 -> park) + AIMD window rebalance to 1x/0.25x capacity (P2 T15) 2026-10-16 08:27:52 +08:00
Khoa Pham a93f16f0cf [Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV (#43922) 2026-10-15 17:21:43 -07:00
Liangsheng Yin d14fa5db9b [Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs (#44598) 2026-10-15 17:20:02 -07:00
Khoa Pham 4cca7fcd3e chore: bump sgl-kernel version to 0.4.10 (#44590) 2026-10-15 15:50:51 -07:00
ec7ab9d0c2 [Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins (#44353)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-15 15:50:25 -07:00
0b1a73599a [Feat] rust-server: Apply the x-override-* request headers (#44312)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 15:50:04 -07:00
William Hu 4e0609ea27 [Fix] moe: Preserve negative routing sentinel in EP mapping (#40498) 2026-10-15 15:05:08 -07:00
Yuwei An 91f6ef2322 [DSv4.1] Fuse request-window KV copies (#44555) 2026-10-15 14:11:51 -07:00
Xinyuan Tong 877be2974a [CI] pr-test: Let stage wait jobs reach their max-wait budget (#44410) 2026-10-15 13:34:49 -07:00
snadampal aa5c7e887c [Feat] docker: Add AWS EFA image variants built on published images (#41006)
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
2026-10-15 13:19:22 -07:00
Ziyi XuandClaude Opus 5.5 bc76a6b819 [DSv4] Sparse prefill: build compressed lengths without a host sync (#44557)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:45:56 -07:00
87211088c9 perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell (#36810)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-15 12:09:09 -07:00
02a43eeef6 [Feat] rust-server: Attach configured metadata headers to a healthy /health_generate (#44315)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-15 12:04:23 -07:00
173e0f2e9b [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode (#44173)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-15 12:02:59 -07:00
Michael bba535ebb4 [Fix] [AMD] docker: Repair the AITER build step after the 2bb549102 pin bump (#44534) 2026-10-15 11:18:41 -07:00
c5a5bf0d0b [Fix] pp: Order output send/recv by rank parity to fix cross-node hang (#44453)
Co-authored-by: chuixue <chuixue@minimaxi.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-16 01:46:37 +08:00
minke.yu 1975265315 gateway g9i: AIMD congestion window replaces slot-estimate admission gate (P2 T14) 2026-10-16 01:33:18 +08:00
Shangming Cai c7fb57d990 [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349 (#44509) 2026-10-16 01:28:57 +08:00
a8aaf95b58 [Cookbook] Add cookbook for muse spark open (#44527)
Co-authored-by: thomawan <thomawan@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
Co-authored-by: Thomas Wang <1am9trash@gmail.com>
Co-authored-by: zijie <zijie.chen@amd.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-15 10:20:38 -07:00
Aleksi Vesantoandjacky.cheng 5ef8319f16 [Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation (#27702)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-16 00:40:41 +08:00
Yihao WangandClaude Opus 5.5 5b12ec163c [Doc] cookbook: Expand the FLUX.3 Image Klein model description (#44515)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 08:56:54 -07:00
Jimmy ShongandJimmy Shong 05cbe241be [Doc] cookbook: Add Nemotron 3.5 Super VL (#44510)
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
2026-10-15 08:39:14 -07:00
28b98e5d13 [Doc] cookbook: Add the FLUX.3 prompt upsampler section (#44512)
Co-authored-by: Sandro Cavallari <scavallari@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 08:38:33 -07:00
minke.yu 534417b82e gateway g9i: D-side in-flight ledger + Little's-law slot release rate (P2 T13) 2026-10-15 23:21:42 +08:00
YC Yen-Ching Tseng e87e32b311 [Feat] [AMD] docker: Release ROCm 10.1 for gfx1250 in parallel with ROCm 10.0 (#42699) 2026-10-15 22:59:54 +08:00
a57707377f [Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware (#44072)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 22:42:33 +08:00
qiaonan02 c4870d98cf [Fix] glm5_next: Correct num_experts_per_tok default to 8 (#44093) 2026-10-15 21:53:31 +08:00
09b7383624 [diffusion] model: support FLUX.3 image (#44418)
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
Co-authored-by: Mick Qian <mickqian@radixark.ai>
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com>
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 21:50:59 +08:00
elvischenvandClaude Opus 5.5 6fe5596ebb Support EPLB for Nemotron-H (#43685)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:50:09 +00:00
ormandjandBenjamin Oldenburg c5bf3aafbc Fix overflow in legacy and v1 DSA top-k selectors (#37625)
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.co.th>
2026-10-15 21:44:58 +08:00
minke.yu ed0a454730 gateway g9i: T_D slot (max_running) dimension — max(memory, slot) wait with dual EWMA (P2 T12) 2026-10-15 21:41:03 +08:00
jiaqiang-dot-liuandCursor 2f5407da28 [Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path (#44266)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-15 21:22:46 +08:00
sogalin_codegen 2092160d42 [AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 (#41988) 2026-10-15 21:08:44 +08:00
amote-i abf92826ee [NPU] [DOC] update NPU best practices and model deployment docs to CANN 9.1.0 (#44432) 2026-10-15 20:22:54 +08:00
Zhang Yiyang (SII)andMick Qian 23c2a026db [diffusion] model: support MOSS-Image (#44325)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 19:23:56 +08:00
minke.yu 04ee2d03d7 gateway g9i: atomic check-and-act admission at decision time (P2 T11) 2026-10-15 18:20:01 +08:00
minke.yu 9d5db13e53 gateway g9i: P-queue hard token cap with candidate fallback (P2 T10) 2026-10-15 17:38:17 +08:00
Cheng Wan f7ad4345b8 [Refactor] deepseek-v4: Use explicit stage boundaries and row transport (#43607) 2026-10-15 02:05:28 -07:00
Cheng Wan 3a835cf417 [Refactor] inkling: Use short-convolution stage boundaries (#43606) 2026-10-15 02:04:46 -07:00
Cheng Wan e1f9fce7b3 [Refactor] layer-boundary: Bind residual completion to producer updates (#43605) 2026-10-15 02:04:02 -07:00
Cheng Wan d59c3448b0 [Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling (#43604) 2026-10-15 02:03:27 -07:00
Eddie Zhou 71a17dbc24 [AMD][CI] setup: Skip redundant pip-cache chown walk and MORI rebuild (#42148) 2026-10-15 16:21:46 +08:00
93dbe88183 Fuse MiMo verification RoPE and KV cache writes (#43106)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 16:15:24 +08:00
kangwangamd 28703575f9 [AMD] Update ROCm AITER pin to 2bb549102 (#44379) 2026-10-15 16:10:41 +08:00
GeorgeSherifandXiaoyu Zhang 8f853b0041 [Doc] cookbook: Add Jais 3 deployment recipes (#42766)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 16:09:44 +08:00
86f4ede4fb [Fix] dsa: Make the fused top-k exact on an overflowing threshold bin (#37941)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: DarkSharpness <2040703891@qq.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 15:52:48 +08:00
9ea5472833 [HiCache] Add the page_unified host KV layout and the unified L3 key (#39727)
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-15 15:46:13 +08:00
33618d4f00 [Feat] runtime: Include FlashInfer’s default tactic in startup tuning. (#43927)
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:30:45 +08:00
Liangsheng Yin a776851c97 [Refactor] scheduler: Check the prefill budget at the admission loop head; drop a redundant PD gate (#44420) 2026-10-15 00:16:56 -07:00
fa66e7d816 [NPU] [Bugfix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism (#29753)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
2026-10-15 09:58:47 +03:00
ashwini rathiandClaude Opus 5.5 01ac2cf36d [CI] [XPU] nightly: Split Intel Main and Wheel workflows (#44108)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:56:27 +08:00
MickandMick Qian f2b73d4f9b [diffusion] doc: document reference resizing, LoRA auto mode and image profiling (#44377)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-15 13:41:44 +08:00
Dayuxiaoshui a50c17e979 [diffusion] perf: fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel (#38650) 2026-10-15 13:39:27 +08:00
676c038aa5 [diffusion] fix: fix the load duration missing from subclassed server tests (#40657)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-15 13:38:23 +08:00
minke.yu 7e20f0af79 gateway g9i: P2 T_D admission gates (off|shadow|enforce) + T_PREFER candidate fallback (T9) 2026-10-15 13:35:50 +08:00
Liangsheng Yin f9bdb74a5a [Fix] scheduler: Count a request's held req_to_token row once at admission (#44401) 2026-10-14 22:29:10 -07:00
Yuxingwang-intel 93842f52fe [CPU]: Add group-aware CPU SHM collective kernels (#35435) 2026-10-15 13:22:20 +08:00
Michael 4dc9b5c68c [Test] [AMD] diffusion: Expect CUDA-only fast paths to be off on ROCm in kernel tests (#44311) 2026-10-15 13:18:01 +08:00
1d809bbbbd [Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails (#43502)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-10-14 22:04:58 -07:00
kkandroot e573d91275 [CI] tests: Fix stale fakes in the CP strategy and request-class override unit tests (#44380)
Co-authored-by: root <root@crsuse2-m2m-223.us-east2-a.compute.internal>
2026-10-14 21:35:20 -07:00
Khoa Pham d4daf52369 [Fix] dsv41: Bound MoE routing to live SWA graph-tail rows (#44347) 2026-10-14 21:14:28 -07:00
AMD-yanfeiwang 08f15e76bb [Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware (#43511) 2026-10-15 12:03:01 +08:00
WenhaoZhangandMick c92108bb93 [diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight (#44058)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-15 11:59:03 +08:00
Liangsheng Yin 3b5ae4ea86 [Fix] scheduler: Drop finished requests from the running batch before every admission (#44351) 2026-10-14 20:40:05 -07:00
2528cd2b91 [PD] Send DSA indexer state only past decode_prefix_len (#44140)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-15 11:25:18 +08:00
Avaya Aggarwalandniehen6174 0cf683ea8d [diffusion] comfyui: return several server-mode images as one IMAGE batch (#44088)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-15 10:38:37 +08:00
Thomas Wang 4851d182a1 [Doc][AMD] Update v41 amd cookbook 1015 (#44368) 2026-10-14 19:33:08 -07:00
xiaofei-zheng a12814c158 [Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk (#43643) 2026-10-15 09:36:16 +08:00
Yuhao YangandClaude Opus 5.5 9fa9308e95 [Doc] cookbook: Add Step 5 Preview cookbook (#44357)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 09:21:06 +08:00
9d68ecce2e [DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits (#44051)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-15 09:00:19 +08:00
xiaofei-zheng c5c4bbbd25 [Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP (#43642) 2026-10-15 08:59:34 +08:00
Michaelandjacky.cheng 8333ccfccb [Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph (#43770)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-14 17:40:07 -07:00
Yuwei An 64a0a08f0a [DSv4.1] Support TRT-LLM decoder bounded replay with BCG (#44326) 2026-10-14 17:39:52 -07:00
YAMYandYuwei An c891c5c7ca [DSv4.1] Fold encoder-SWA replay into the extend forward (#43166)
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-10-14 17:27:07 -07:00
Shuwen Wang 3c28614f12 [Fix] hicache: Defer query beliefs until the rank-synchronized hit drain (#43108) 2026-10-14 17:25:08 -07:00
Aditya Singhandhnyls2002 a6a8c56015 [Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config (#33715)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 17:20:21 -07:00
0536cec73d [Feat] metrics: Expose multi-detokenizer router CPU usage (#43323)
Co-authored-by: raghul chandramouli <raghul@c1xind0180-0243.local>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 17:17:22 -07:00
Junnan Liandhnyls2002 0f36c8015c [Bug] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields (#31892)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 16:56:56 -07:00
209f8220d6 [Fix] sglang-server: Honour extra_key on the Rust frontend's /generate (#44207)
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:44:31 -07:00
e7d6a88bfc [HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows (#43932)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: houseroad <houseroad@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-14 16:43:24 -07:00
aa8793f33b [Rust] Serve scheduler metrics on the Rust frontend's /metrics (#43942)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:43:06 -07:00
2a9f6a25a7 [Core] Add subclass hooks for the generation request class and optimistic prefill eligibility (#44113)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: CuiCoco <CuiCoco@users.noreply.github.com>
2026-10-14 16:42:30 -07:00
5f2861bf59 [Perf] session: Cache multimodal prompt token counts across turns (#44112)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-14 16:41:34 -07:00
9732188a58 [Rust] Serve /weights_checker from the Rust frontend (#43926)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-14 16:41:10 -07:00
FREETRUMPandhnyls2002 97d3150a6a [Perf] Cache msgspec struct fields in the batch splitter (#43520)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-14 14:51:26 -07:00
4273814771 [Perf] detokenizer: Group return IPC sends by tokenizer worker (#44288)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-14 14:26:51 -07:00
Charles Chenandalphabetc1 d3dc14eb38 [mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache (#40857)
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-10-14 14:19:51 -07:00
Rain Jiang 12dc623102 [Chore] rust: Remove the sglang-renderer crate (#43451) 2026-10-14 11:43:37 -07:00
Shuwen Wang b67785153a [Fix] glm5.2: Align HiSparse warmup backends and move test to nightly (#44303) 2026-10-15 01:28:03 +08:00
Yoray Zack 87ed1b9526 [4/N] elastic-ep: Scale at DP-attention replica granularity (#33728) 2026-10-15 01:17:01 +08:00
c964d9ba20 [Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys (#42818)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-15 00:50:19 +08:00
amote-i 3a9cae27ed [Doc] [NPU] docs: Update NPU CANN version to 9.1.0 (#44177) 2026-10-15 00:27:18 +08:00
Michael 1737251730 [Test] [AMD] diffusion: Run the IPC all-to-all transport tests only on CUDA (#44096) 2026-10-14 23:04:23 +08:00
010776e94f [KV events] Publish opaque LoRA block IDs and adapter names (#42817)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 22:56:28 +08:00
Xinyuan Tong ed40c7fc66 [Doc] cookbook: add Ling-3.1-flash (#44301) 2026-10-14 22:40:01 +08:00
minke.yu 0b74ab2ac4 gateway g9i: shadow-consistent routing switch --g9i-shadow-route (P1 shadow T7) 2026-10-14 22:22:58 +08:00
minke.yu 3c7ec33d26 gateway g9i: est_T_D shadow metrics — est_tp/est_td histograms, residual by dominant component, pool gauges (P1 shadow T5) 2026-10-14 21:52:07 +08:00
minke.yu 09ca4957ee gateway g9i: scrape P /metrics kv_transfer_bootstrap_ms as est_T_D ground truth (P1 shadow T4) 2026-10-14 21:50:11 +08:00
minke.yu 7921442a5c gateway g9i: est_TTFT max(T_P, min_D T_D) composition with C4/C13 degradation (P1 shadow T3) 2026-10-14 21:44:17 +08:00
minke.yu 05ef104208 gateway g9i: pool_free_rate completion EWMA per decode node (P1 shadow T2) 2026-10-14 21:34:56 +08:00
minke.yu d44620f285 gateway g9i: fetch disagg/queues sections in /v1/loads, extend NodeLoad for est_T_D (P1 shadow T1) 2026-10-14 21:21:11 +08:00
Kan WuandClaude Opus 5.5 d2a436446c [Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models (#44252)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:21:01 +08:00
204644c2dc [Fix] sglang-processor: Follow Python's OpenAI layer through #43027, #37577, #43006, #43026, #43005 (#44243)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 21:06:13 +08:00
triple-mu 5e90312eb1 [diffusion] perf: support bit-exact norm/bias fusions and channels_last decode for Qwen-Image 2.1(#41340) 2026-10-14 20:47:46 +08:00
Shuwen Wang 7b53a13401 [Fix] glm5.2: Align PD HiSparse prefill KV layout (#44256) 2026-10-14 20:40:28 +08:00
bab9c81fea fix: FA4 paged-KV zero-size page-table registers on SM90 (#35757)
Co-authored-by: razorback16 <razorback16@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-14 19:10:15 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 3b72b70ef4 [CI] vlm: Pin baidu/Unlimited-OCR revision in test_unlimited_ocr_server (#44235)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:50:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad c82e8bd5fe [Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode (#44210)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-14 17:31:19 +08:00
df1a089cd3 [Spec] Fix DFLASH verify mask policy for wrapped attention backends (#43054)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-14 17:15:36 +08:00
817781b828 [Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor (#43689)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-14 17:09:35 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 72730d6360 [Doc] cookbook: Mirror SA cluster NCCL_CUMEM_ENABLE in AgentX launches, fix GPU counts (#44228)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:07:40 +08:00
c54ba66cd4 [Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo (#43639)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 15:39:44 +08:00
Mohammad Miadh AngkadandMohammad Angkad 6daae9c98e [Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted (#44023)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-14 00:01:32 -07:00
58f2c923ed [Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers (#44218)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
2026-10-14 14:59:06 +08:00
Kan WuandClaude Opus 5.5 53a24c448e [Feat] sgl-router: Render Kimi-K3 through sglang-processor (#43608)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:40:18 -07:00
Michael 6f646563a2 [AMD] diffusion: expect the residual-gate fast path to be off on ROCm in test_modulate (#40660) 2026-10-13 23:34:31 -07:00
Ma Mingfei 68c18a5701 Revert "[XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend" (#44216) 2026-10-14 14:27:46 +08:00
desmond-intel b87d62e581 [XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend (#39805) 2026-10-14 14:24:04 +08:00
Shunkangz 0d130f1915 [Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8 (#39724) 2026-10-13 23:19:46 -07:00
7ca4cbfcef [Disagg] Return prompt routed experts from the PD prefill server (#44002)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-13 23:13:44 -07:00
AndyLi429andAndyLi429 05bb5afb62 [NPU][Bugfix] use CANN libdevice for DSpark Markov sampling (#43849)
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com>
2026-10-14 14:12:22 +08:00
Zhaoyi Li e29eee18df [AMD][DI][CI] nightly mi355x disagg: stage packed logs under RUNNER_TEMP, not /tmp (#43945) 2026-10-14 13:53:40 +08:00
Kan WuandClaude Opus 5.5 4560da58ee [sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor (#43625)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:19:44 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 d5f80aa3c3 [Test] dsv4: Fix Hopper indexer replay test after DeepSelect decode (#41561) (#44193)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:15:35 +08:00
ANSHUMAN TRIPATHY 30bdd09aed DSPARK enable for XPU (#33739) 2026-10-14 13:10:24 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 721953c6f6 [Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main (GLM-5.2 NVFP4 B300 agg TP4, Dynamo frontend counts) (#44065)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:03:51 +08:00
Khoa Pham e00754bab1 [DSV4.1] Derive the late-layer DP geometry from the captured shape under the breakable prefill graph (#44144) 2026-10-13 21:54:35 -07:00
Mohammad Miadh AngkadandMohammad Angkad ff8d964cf5 [CI] glm5.2: Temporarily disable PD HiSparse+MTP B300 test (#44192)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 21:53:58 -07:00
chenxb002andchenxiaobing 491d4ef178 [MLU][CI] Bootstrap MLU runner workflows (#41005)
Co-authored-by: chenxiaobing <chenxiaobing@cambricon.com>
2026-10-13 21:40:49 -07:00
Chetan Kumar VermaandClaude Opus 5.5 2c1e85964e [diffusion] cache-dit: respect CFG parallel for dual-transformer models (#43704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 12:25:07 +08:00
xinchen9andMa Mingfei 2428d0f531 [XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba… (#42122)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-10-14 12:10:12 +08:00
16176278d0 [diffusion] perf: replay small Triton wrappers' launches per operand signature (#43859)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:18:19 +08:00
Mohammad Miadh AngkadandMohammad Angkad 4f751da975 [Agents] Add a rule to prefer sgl-eval for accuracy evals (#44040)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 20:17:04 -07:00
53fe21cdfd [diffusion] perf: replace padded rows without a host sync for Z-Image (#44069)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:15:28 +08:00
xiaofei-zhengandXiaoyu Zhang 7e226b1b71 [AMD] Use the preallocated EAGLE topk=1 draft chain on HIP (#43640)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 11:06:07 +08:00
5efb7384c1 [diffusion] comfyui: consolidate 6 unrelated plugin fixes (#43493)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-14 10:56:09 +08:00
Cheng Wan c053fc71d3 [CI] pr-title: Validate pull request title format (#44165) 2026-10-13 19:53:52 -07:00
Ziyi XuandClaude Opus 5.5 4f5fcbfef5 [JIT] Clean up kernel headers: move runtime helpers, add once.h (#41147)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 10:46:51 +08:00
yuyu5333andClaude Code b50fdeb0ef [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. (#41561)
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-14 10:46:12 +08:00
Ma MingfeiandMingxuZh 87beb3a052 [CPU] Temporarily disable subblock sparse attention test on Xeon CI (#43468)
Co-authored-by: MingxuZh <109504044+MingxuZh@users.noreply.github.com>
2026-10-14 10:44:24 +08:00
13a6dee99b [Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16 (#42837)
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-14 10:32:50 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7f0434e4af [GLM-5.3-Flash] Drop the NextN layer from the KDA state pool (#44098)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 19:31:02 -07:00
Avaya Aggarwal ac187c0cd5 [diffusion] Bound frame_interpolation_exp to RIFE's supported range (#43014)
Signed-off-by: onepunchmonk <aggarwal.avaya@yahoo.com>
2026-10-14 10:29:28 +08:00
JinandXinyuan Tong 44b44774c3 [Bugfix] Keep rope_theta when --json-model-override-args overrides rope_scaling (#43067)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-14 10:16:35 +08:00
Baizhou Zhang 536d065370 [CP] Automatically enable BCG for supported prefill CP configurations (#44022) 2026-10-13 19:00:42 -07:00
shyeh25andClaude Opus 5.5 684fea8af3 [DSV4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches (#43993)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:36:15 -07:00
Liangsheng Yin 2ebdf9360b [Fix] Work around a CPython fork deadlock in the CPU preprocess pool (#44143) 2026-10-13 16:31:57 -07:00
b3991b53da GLM5.2 DSA Indexer Fallback (#32947)
Co-authored-by: Shu Wang <shuw@login-hecate03.hecate.clusters.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 16:27:33 -07:00
Khoa Pham 5917604a95 [DSV4.1] Support decoder SWA bounded replay with CUDA DP attention (#43812) 2026-10-13 16:10:57 -07:00
HEJIAN SANGandClaude Opus 5.5 a832609b92 [Feature] Return output top logprobs as flat base64 arrays on request (#42890)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:58:31 -07:00
baa632b049 [Score] Send stacked GPU embedding overrides through a CUDA IPC pool (#39730)
Co-authored-by: Sundara Raman Ramachandran <sundar24295@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-13 15:46:50 -07:00
Andrew Briand 08550acc28 Mark FP32 redux asm volatile (#44137) 2026-10-13 15:08:52 -07:00
Jiaqi LiandZijie Xia 5569b14d26 fix(benchmark): request token IDs when counting OpenAI prompts (#34511)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-10-13 14:31:45 -07:00
Yizheng Jiao 2341e647d3 [DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph (#43736) 2026-10-13 13:55:09 -07:00
79af3df346 [gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request (#42673)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:46:15 -07:00
huangtingwei 18210ed849 [HiSparse] Add speculative decoding support with multi-step KV cache swapping (#37771) 2026-10-13 13:38:31 -07:00
Xinyuan TongandNihar Patel 7dc2930585 [Feature] Add Aaronson-Gumbel text watermarking (#37577)
Co-authored-by: Nihar Patel <nihpat95@gmail.com>
2026-10-13 12:47:50 -07:00
Xinyuan Tong 5eda6b7e67 docs: add MiniCPM-V 4.7 cookbook (#43829) 2026-10-13 12:04:47 -07:00
ce5352a814 [Rust] Serve any model name on the OpenAI chat and completions routes (#43923)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:58:57 -07:00
01f3ed18a5 [unified memory pool] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode (#43919)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-13 11:58:24 -07:00
7e3da1b784 [HiCache] Extract the buffer-mode prefetch and load-back lifecycle into pipeline operations (#43870)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:51:44 -07:00
5bcaf0f8ed [Core] Add request-level customized_info keys and mark Full prefill capture batches (#43772)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-13 11:45:52 -07:00
149ef7d1b7 [Rust] Let an embedding package wrap the frontend HTTP app (#43924)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:44:43 -07:00
a8381d73be [HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction (#43868)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:39:48 -07:00
Khoa Phamandvincentzed 8bf145d119 [DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP (#43735)
Co-authored-by: vincentzed <207368749+vincentzed@users.noreply.github.com>
2026-10-13 11:35:34 -07:00
6ade23696d [Core] Keep request priority within the signed 64-bit range in radix caches (#43920)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: qandrew <qandrew@users.noreply.github.com>
2026-10-13 11:33:26 -07:00
7f9f742535 [Rust] Refuse an unsupported multimodal model before the Rust listener binds (#43925)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-13 11:30:43 -07:00
2a8e9db3b0 [lora] Support LoRA for the NemotronH_Nano_VL_V2 model family (#39798)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-10-13 11:29:48 -07:00
ff3f8c348c [Observability] Report per-request storage prefetch and staging waits in meta_info (#43946)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-13 11:13:01 -07:00
Rain Jiang e98c6d7b25 Rainj me/rust bench (#43950) 2026-10-13 11:07:42 -07:00
Kailash RanganathanandYanbin Jiang c0584cf9af [perf] [multi-lora] remove empty CTAs from fused LoRA expand and add split-K to shrink (#41951)
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-10-13 10:16:14 -07:00
Zhaoyi Li e9400c5a8f [AMD][DI][CI] kimik26 2p1d-ep16: capture CUDA graph on decode (eager decode is slow) (#43985) 2026-10-13 08:46:53 -07:00
Zhaoyi Li 96f519e4fa [AMD][DI][CI] glm52 2p1d-ep16: capture CUDA graph on decode (eager decode is too slow) (#43977) 2026-10-13 08:45:46 -07:00
11cd2701a8 fix(openai): preserve zero cached tokens when cache reporting is enabled (#43005)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 23:29:19 +08:00
3f1e185f6e fix(openai): keep chat stream created timestamp stable (#43026)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 22:43:33 +08:00
jiaqiang-dot-liuandCursor 4c5ed0597e Skip fully-masked KV tiles in sliding-window extend attention (#39769)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-13 22:40:53 +08:00
27b68303d0 fix(openai): omit absent tool_calls from chat responses (#43027)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-13 20:56:58 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 8e37c6cf37 [CI] Run test_pi05_modelopt_fp8_cuda in the kernel-unit lane (#44055)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:39:02 +08:00
Ho-Ren (Jack) ChuangandXiaoyu Zhang 5100f8d11d [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache (#41790)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-13 20:30:13 +08:00
amote-i 29974d543e [NPU] [DOC] update kimi k3 cookbook and supported features on NPU (#44052) 2026-10-13 20:27:26 +08:00
jlqibmandClaude Opus 5 74e0a47863 Fix/compressed tensors nvfp4 hopper (#40022)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-13 20:10:31 +08:00
LittleNyimaandMick Qian 5727f08932 [diffusion] feat: add configurable reference image resizing for MiniMax H3 Ref2VA (#35839)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 19:48:18 +08:00
WenhaoZhang 85254b5a8f [diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode (#43581) 2026-10-13 19:47:02 +08:00
Yunhao Zhouand周云昊 991a29d0ad fix(multimodal): convert grayscale tensors to RGB in PIL backend (#42127)
Co-authored-by: 周云昊 <zhouyunhao@zhouyunhaodeMacBook-Pro.local>
2026-10-13 19:03:48 +08:00
4b5b727145 Fix EVS for NemotronH_Omni_Reasoning_V3 (#42933)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 10:59:00 +00:00
minke.yu 5df4f2c036 DEBUG-ONLY: use crc32 for rid fingerprint (builtin hash() is per-process randomized) 2026-10-13 18:12:56 +08:00
minke.yu db1e8cdbe3 DEBUG-ONLY: ARDBG batch fingerprint + phase + hicache sync logs, sampled device AR (1/1024) + idle spin sampling (p2 wedge trap) 2026-10-13 18:01:36 +08:00
MichaelandClaude Opus 5.5 aa15b06abe [AMD][CI] Run test_tracing.py in the extra tier, matching CUDA (#43928)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 02:45:58 -07:00
Xiaoyu Zhang 4e0fa52530 [Perf] Compact GDN causal-conv grids for skewed prefill batches (#43570) 2026-10-13 17:45:06 +08:00
Ziang Li 01f6840693 [FlashInfer] Add startup autotuning for MegaMoE (#43789) 2026-10-13 02:30:31 -07:00
Mohammad Miadh AngkadandMohammad Angkad 195cb7afb4 [CI] Use sgl-eval for GLM-5.3-Flash GSM8K tests (#43975)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-13 02:26:50 -07:00
Shu WangandAnish Shanbhag e0c957e222 [HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation (#41769)
Co-authored-by: Anish Shanbhag <ashanbhag@nvidia.com>
2026-10-13 02:07:54 -07:00
Shu WangandPo-Han Huang 27be99ae73 [NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe (#40772)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-13 02:00:24 -07:00
minke.yu f3ac513d92 DEBUG-ONLY: limit ARDBG stack logging to ONE_SHOT_PUSH calls (volume control) 2026-10-13 16:01:18 +08:00
minke.yu 0054c5b8fe DEBUG-ONLY: SGLANG_AR_SEQ_DEBUG seq logging for TP device AR + host CP gloo AR (p2 wedge repro) 2026-10-13 15:52:17 +08:00
minke.yu 6da0a40b02 g9i shadow: fix temporary-borrow compile errors in new tests 2026-10-13 15:39:06 +08:00
minke.yu caf3403086 g9i shadow: fix cached_tokens_details placement to sglext container
Verified against serving_chat.py/protocol.py on this branch: the engine
emits cached_tokens_details under top-level "sglext" (SglExt), omitted
entirely when all fields are None; ids-framed streams emit a standalone
choices=[] sglext chunk with no usage. extract_cached_breakdown now reads
sglext first (usage/top-level kept as fallbacks); strip_cached_details_from
removes the field from sglext and drops the emptied key; process_sse_chunk
pre-filters on "sglext" too and drops events that only carried stripped
fields. Adds 5 placement tests.
2026-10-13 15:33:03 +08:00
minke.yu 617f19ba61 gateway tests: fix PDRouter initializer missing g9i field (pre-existing test compile break) 2026-10-13 15:18:33 +08:00
Chunan ZengandClaude Opus 5.5 7ff3a11284 [Perf] Enable fast-math for JIT activations on Blackwell (#43236)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:18:05 +08:00
minke.yu ba3318ffbd gateway g9i: fix shadow compile errors — lookup returns owned snapshot (borrow across loop mutation), bind temp in test 2026-10-13 15:14:05 +08:00
minke.yu 8279d3d6df gateway g9i: v2 P1 shadow mode — prefix-hash chain, bookkeeping table, shadow classifier + queue model
Observability only, zero routing influence, gated by --g9i-shadow-enabled
(requires --g9i-enabled). Design: experiments/g9i/g9i-v2-design-20261013.md.

- g9i_prefix_chain.rs: rolling xxh3 chain over chat messages (seed folds
  model/template-kwargs/tools/system; tool_calls/tool_call_id/name hashed;
  byte-level image semantics; unclassifiable flag for unknown parts)
- g9i_shadow.rs: PrefixTable (LRU+TTL, deepest-hit lookup, exact tail
  anchor + placeholder middle boundaries), ShadowClassifier (hot/short/
  cold_image/cold_large/cold_unknown via byte heuristic), QueueModel
  (per-(node,class) outstanding + completion-driven drain EWMA, est_TTFT),
  usage/cached_tokens_details injection+strip pairing
- pd_router.rs: request-side shadow classify + agreement counters (shadow
  vs v1 probe verdict), response-side usage harvest on stream/non-stream,
  queue admit/complete/release conservation incl. abort paths
- main.rs: --g9i-shadow-enabled/-table-max-entries/-table-ttl-secs

Not yet compile-verified (no local toolchain); b300 container check next.
2026-10-13 15:05:56 +08:00
AMD-yanfeiwang d8f8522348 [router] power_of_two: in-flight request load for --dp-aware decode (#43483) 2026-10-13 14:50:54 +08:00
Michal ShalevandPo-Han Huang 35a014c7e8 [PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids (#43723)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-12 23:48:13 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 dbad07a06f [Docs] Cookbook: refresh SemiAnalysis AgentX launches (DeepSeek-V4.1 GB200/GB300, GLM-5.2 FP8 B300) (#43983)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:39:45 +08:00
50a144057c [AMD][PD] enable mori transfer for PD decode DCP (#37358)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-13 14:35:09 +08:00
Xiaoyu Zhang 46b7357779 [Diffusion] Speed up LTX-2 guidance and video output (#43667) 2026-10-13 14:23:33 +08:00
Jimmy Shong 3eebf987ed [PD] Support heterogeneous TP for hybrid SWA KV transfer (#40909) 2026-10-12 23:21:54 -07:00
603a05e6c0 [ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP (#43328)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-12 23:17:49 -07:00
AMD-yanfeiwang 17c3b259f5 [router] Fix power_of_two with --dp-aware (#42027) 2026-10-13 14:15:36 +08:00
Zhaoyi Li fc404bbbb3 [AMD][DI][CI] dsv4pro-fp8 2p1d-ep16-mtp: raise prefill mem fraction for MORI EP v2 memory (#43944) 2026-10-12 23:03:57 -07:00
Xinyi Song a69e893b1a [AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1 (#42911) 2026-10-12 22:33:53 -07:00
jacky.chengandMick Qian 42e43cf33f [AMD] Fix Flux2 kv-gather abort on non-contiguous K/V (#43793)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 22:20:55 -07:00
kkandwunhuang 43356304b4 [AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels (#43968)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-12 22:09:55 -07:00
WenhaoZhangandMick Qian ea13359da9 [diffusion] perf: implement paint_use_remesh (decimate to 40k faces before UV unwrap) for Hunyuan3D (#42770)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 12:28:32 +08:00
6a89181614 tokenizer perf optimization (#39771)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-13 12:27:21 +08:00
WenhaoZhangandMick Qian feba04fa49 [diffusion] fix: avoid redundant LoRA scaling while preserving exact numerics (#43028)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-13 12:27:04 +08:00
WenhaoZhang cda267ab0f [diffusion] fix: keep MiniMax-H3 VAE fused norms off ROCm (#43350) 2026-10-13 12:26:16 +08:00
WenhaoZhang b4822bae03 [diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode (#43504) 2026-10-13 12:01:48 +08:00
4d5692b6cd fix(openai): honor tool_choice=none in DeepSeek encoders (#43006)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 11:41:50 +08:00
Piotr MazurekandCodex 40f5d9411f Reuse Hugging Face LFM2-MoE configuration (#43939)
Co-authored-by: Codex <codex@openai.com>
2026-10-13 11:29:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7bd41aec12 [CI] Remove Mistral Small 4 e2e test (#43973)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-12 20:14:22 -07:00
Thomas Wang c2dfd5d3d6 [AMD] Update v4 amd cookbook 1012 (#43962) 2026-10-12 19:57:55 -07:00
Xiaoyu Zhang 03416e8c0f [Fix] Account for captured inputs in DeepGEMM masked workspace budget (#43563) 2026-10-13 10:53:10 +08:00
Liangsheng Yin 43e4e84731 [PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode (#43936) 2026-10-12 19:32:03 -07:00
Yuwei An 07566cbf0d [DSv4.1] Temporarily disable B300 PD/CP accuracy test (#43948) 2026-10-12 19:31:10 -07:00
75a0d91c1f [Fix] Preserve HTTP errors for non-streaming /v1/messages (#43796)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-13 10:10:45 +08:00
45418d7dc6 [HiCache] Avoid full-store scans for uncached file prefix lookups (#42077)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-12 18:52:50 -07:00
Yuwei An 23150f7ae0 [DSv4.1] Reduce sparse-prefill workspace padding (#43921) 2026-10-12 18:51:36 -07:00
Ziyi XuandClaude Opus 5.5 107cff72d3 [DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows (#43833)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:45:16 -07:00
e775fa7253 [Build][CI] Drop Python 3.10: require Python 3.11+ and run CUDA CI on Python 3.12 (#43021)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-10-12 18:40:16 -07:00
YAMY 59445b3da1 [DSV4][CP] Write low-ratio page indices when CP prefill runs paged attention (#43935) 2026-10-12 18:36:11 -07:00
Michael 7224fce1b5 [AMD] Recalibrate test_quark_mxfp4 GSM8K thresholds to observed MI355X noise (#41980) 2026-10-12 16:46:03 -07:00
Yizheng Jiao 6a7abcfc9b [DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD (#43734) 2026-10-12 15:32:17 -07:00
Khoa Pham 93809202a3 [Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs (#43918) 2026-10-12 14:50:04 -07:00
Duyi-Wang 6781da8e5b [AMD] [Docker] Update MoRI to v1.2.4 (#43834) 2026-10-12 14:45:42 -07:00
Thomas Wang 64cffed139 [AMD] Fix test_compact_attention_hip.py CI (#43836) 2026-10-12 14:43:19 -07:00
Thomas Wang d6853f5066 [AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill (#43803) 2026-10-12 14:42:02 -07:00
Michael 6c2a682b59 [AMD] Keep the FP4 varctx schedule test off AITER's 2048-row compile (#41793) 2026-10-12 14:39:08 -07:00
Zhang, JiejingandJohnQinAMD 6966c5d027 [ROCm] Eliminate HiCache migration stalls and use exact-size host allocations (#42553)
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: JohnQinAMD <yanyuan.qin@amd.com>
2026-10-12 14:05:51 -07:00
ix-48 b60444c621 Fix autograd retention in CUDA graph capture tail hooks (#43854) 2026-10-12 12:31:18 -07:00
Liangsheng Yin 0efbb3d2b9 [DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept (#43791) 2026-10-12 11:35:57 -07:00
Liangsheng Yin 392039d752 [DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager (#43790) 2026-10-12 11:35:41 -07:00
minke.yu ef4cccb6a0 gateway g9i: allow flow-control-only mode (no fast-P URL) — B case flat pool 2026-10-13 01:31:57 +08:00
Shuwen Wang f92199b569 [Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler (#40896) 2026-10-13 00:03:21 +08:00
minke.yu 98b950366d mooncake PD transfer: mix bootstrap_room into lane shard hash (fixes 1/16 lane collapse with single decode instance) 2026-10-12 23:41:23 +08:00
Sam ShleiferandClaude Fable 5 d88d5c325a Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init (#37525)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-12 23:27:21 +08:00
Yuan Luoandluoyuan.luo 23a1f53fc3 [QSA] Bound the decode indexer's work by the context length (#43463)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-10-12 22:47:45 +08:00
AlexandrandАртем Савкин 1a553927b0 [CI][Diffusion] Add caching for npu GT images in CI (#37426)
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
2026-10-12 17:18:01 +03:00
279f1fad90 [diffusion] chore: bind worker CPUs to the GPU's NUMA node when memory policy is not permitted (#43756)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:09:41 +08:00
minke.yu 87ab6c0155 gateway g9i: probe fast-P only + L3-inclusive classification + exact-body probe cache (A案; removes accidentally reintroduced home_hit) 2026-10-12 13:03:54 +00:00
minke.yu 0308920bdb gateway g9i: retry probe once on transport errors (stale keep-alive pool)
Observed twice on 2026-10-12: reqwest reused server-closed pooled connections,
all-candidate probes failed with connect-level errors, fail-closed 429 drowned
hit-class traffic. Retry once on is_connect/is_request (ms-scale reconnect);
timeouts stay non-retried (wedged engine + 6s worst-case TTFT budget).
2026-10-12 12:02:49 +00:00
Артем Савкин d1357bf87b [NPU] Diffusion CI fixes (#43210) 2026-10-12 14:53:51 +03:00
ming_wang c6e822c29b [Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change (#43309) 2026-10-12 19:24:05 +08:00
Sasha SidorovandShangming Cai 7953751bfa [PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution (#41783)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-12 18:49:52 +08:00
8afe107c24 [XPU] Upgrade XPU backend to PyTorch 2.14 (#42003)
Co-authored-by: Ashwini Rathi <arathi@habana.ai>
Co-authored-by: Y <ashwini.rathi@intel.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-10-12 18:24:48 +08:00
minke.yu 57db2a4ae9 gateway g9i: fail-closed 429 on probe failure / fast-P snapshot loss (p2 incident)
User directive 2026-10-12: 先严格429. When p2 hung, the fail-open fallback dumped
all classified traffic into the regular pool and drowned it (7.5M pending tokens).
All probe-failure paths now reject: prefill_probe_failed (429) when no candidate
can answer /query_cache_hit; fast_p_no_snapshot (429) when snapshots go stale
between probe and pick, with boot grace (had_any_success) mirroring the D gate.
2026-10-12 10:16:09 +00:00
CandCheng Wan 28621739fc [Fix] Make tokenizer request-state initialization atomic (#41911)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-12 03:10:57 -07:00
180933518e [sgl-router] Serve the Anthropic Messages API on /v1/messages (#42316)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-12 17:56:08 +08:00
Jinyan Yi f006fb49ec [NPU] Stabilize HiCache and registered CI tests (#43308) 2026-10-12 17:10:26 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 b0c84d4518 [Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) (#43799)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:08:11 +08:00
Baizhou Zhang 450837e770 [Docs] Update GLM-5.3-Flash playground: B300 default, CP8, EP and PD (#43802) 2026-10-12 01:38:12 -07:00
BingjiaWang 62ff466ac0 [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (#40323) 2026-10-12 01:05:21 -07:00
BingjiaWang 54409c2fa0 [BCG][DSV4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs (#38603) 2026-10-12 01:03:10 -07:00
Anupa SajikumarandSiju Samuel 8d491fbc3e [XPU] Support decode context parallelism (DCP) on Intel XPU (#34355)
Co-authored-by: Siju Samuel <siju.samuel@intel.com>
2026-10-12 15:50:33 +08:00
Ke Bao a70a83bcd2 Keep NIXL Mamba checkpoint keys separate by TP rank (#43594) 2026-10-12 15:40:59 +08:00
Ke Bao b6ba148384 Capture KV host transfer buffers (#43711) 2026-10-12 15:40:06 +08:00
Mohammad Miadh AngkadandMohammad Angkad 06887827c7 [CI] Move tests that do not need GB300 off the per-PR GB300 suite (#43784)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-12 00:38:10 -07:00
5ca0695aa6 [HiCache] Preserve all MTP KV and recurrent state (#40223)
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-12 15:37:59 +08:00
Shuwen Wang f5f5003323 [HiCache] Support Mamba in buffer-only host memory mode (#36345) 2026-10-12 00:35:16 -07:00
Shangming CaiandClaude Fable 5.1 4195980f5a [PD] Poison the deferred abort ack when a Mooncake send fails (#43719)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 15:33:09 +08:00
da4bf1ef8e [Perf] Route BF16 mixed-batch decode tails through decode attention (#43238)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-12 15:27:22 +08:00
Xinyuan Tong a0e6effe50 [Fix] Link cuda_ipc JIT against CUDA driver stubs dir (#43764) 2026-10-12 15:26:32 +08:00
Baizhou Zhang 62c853b654 docs: clarify prefill CP terminology and simplify wording (#43787) 2026-10-12 00:24:57 -07:00
208df292e5 [diffusion][xpu] FLUX.2: enable the bit-exact Triton SwiGLU on XPU (#41940)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 15:16:58 +08:00
Mohammad Miadh AngkadandMohammad Angkad 166b4ae7fc [Fix] Adapt to transformers 5.18 / 5.19 changes (#43688)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 23:24:57 -07:00
ashwini rathi 8ce76b88df [XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast (#42048) 2026-10-12 14:24:26 +08:00
Mohammad Miadh AngkadandMohammad Angkad 130e11d50e [Deps] Bump FlashInfer to 0.7.1 (#43494)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 23:23:11 -07:00
sglang-botandsglang-bot f9bd83e7a9 docs: sync LMSYS SGLang blog cards (#43462)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-12 06:07:15 +00:00
Liangsheng Yin 9e0b001db4 [CI] Skip eval retries in the PD transfer-failure test (#43769) 2026-10-11 22:37:40 -07:00
Liangsheng Yin 2240049194 [RL] Return a failed weight update when the cache flush is refused (#43758) 2026-10-11 21:57:58 -07:00
a297cd3da1 [Fix] scheduler: stop latching batch_is_full when nothing can run (#41872)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-11 21:56:19 -07:00
Mohammad Miadh AngkadandMohammad Angkad bf703013f2 [Test] Re-anchor the VLM flashinfer throughput floor (#43762)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-11 21:34:50 -07:00
d434ea444d [DSV4.1] Reduce eager Engram extend preparation (#42159)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:31:55 -07:00
Liangsheng Yin a1001bcf3a [PD] Retract transferred decode requests on a retract pause (#43757) 2026-10-11 21:29:39 -07:00
ChengYM 0d0be8fc98 [AMD][MORI] Add MORI EP V2 support (#40666) 2026-10-11 20:54:13 -07:00
Shunkangz 20c49f12c9 Support kv cache sharding for DSA indexer (#40925) 2026-10-12 11:50:23 +08:00
37d2e5c34f [AMD][V4.1][*/N] Fix DP attention + TP MoE decode NaN and support bounded replay (#43077)
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-11 20:47:13 -07:00
weireweire 296774da37 [Tokenizer] Require fastokens>=0.3.2 and include it in the all extra (#43104) 2026-10-11 20:41:53 -07:00
adenzhou1350 cd8ed1bb55 [Bugfix] Preserve available host memory for integrated MUSA devices (#43398)
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
2026-10-11 20:34:58 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 6ac8602e8d [CI] Fix TestQwenImageModulationCache after Qwen-Image modulation sharding (#43747)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:31:29 -07:00
77a15cf4e2 [MemCache] Add shared component size and path APIs to the tree core interface (#43722)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-11 20:14:07 -07:00
95ce9caaf2 [Rust] Lock hyper 1.12.0 and want 0.3.2 in the rust workspace (#43724)
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-11 20:12:55 -07:00
e4cfc0c42a [MemCache] Split the SWA mapping move out of locked-full recovery as RecoverEvictedSWA (#43721)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-11 20:11:52 -07:00
WangHemeng 0c52cb1519 [MoE] Skip block-padding rows in the TMA down activation and quant (#42212) 2026-10-12 11:10:30 +08:00
Alison Shao 1a7af3290d Fix Mistral-native format detection when the Hub is unreachable (#43446) 2026-10-11 19:59:19 -07:00
Chunan Zeng 3c74f68a9c [Fix] Initialize BF16 GEMM dispatch in the one-batch benchmark (#43702) 2026-10-12 02:53:02 +00:00
minke.yu 7fd7fd58a5 gateway g9i: hit class always routes to fast-P pool (命中都在 fast-P 发生)
User directive 2026-10-12: read-your-own-node home routing sent hit-class
requests back onto the regular-P long-prefill queue (150s p50 observed),
defeating the fast path. Hit class now always goes to the fast-P pool
(least-wait aggregate); fast-P resolves the prefix from its own L2 or the
shared L3 (wait_complete), recompute-and-graft otherwise — self-healing
land-and-stay. Home override removed; fast-P overload is handled only by
the fast_p_queue 429 gate. New decision counter routed_fast_p_from_regular
distinguishes L3-fetch/recompute hits from local-L2 hits (observability).
2026-10-12 02:20:12 +00:00
WenhaoZhangandMick Qian d0a098b2e0 [diffusion] perf: split and batch LTX-2.3 one-stage guidance passes under CFG parallel (#43684)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-12 09:36:01 +08:00
WenhaoZhang a954b395e6 [diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder (#43683) 2026-10-12 09:34:35 +08:00
WenhaoZhang a67228aa43 [diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP (#43691) 2026-10-12 09:32:52 +08:00
minke.yuandKimi 71a72b8fea gateway g9i: keep large prefills off fast-P even without least-wait
regular_plan returned Default when regular_pool_least_wait=false, letting
the router policy pick across ALL prefill workers including the fast-P
node — ~1/3 of large prefills landed on fast-P under random policy
(observed 2026-10-12), polluting its small-prefill queue and L2. With a
fast-P configured, 'regular' now explicitly restricts to the regular
pool via the least-wait aggregate pick. Default is kept only when the
fast pool is empty (then it equals the regular pool anyway).

Co-authored-by: Kimi
2026-10-12 08:58:03 +08:00
minke.yuandKimi 6c9518fcd8 gateway g9i: move ProbeHit to module scope (fix compile)
Co-authored-by: Kimi
2026-10-12 08:20:30 +08:00
minke.yuandKimi 3a4ab451ac gateway g9i: classify on local (L1+L2) hits only, probe the whole pool
Follow-up to the home-hit routing fix. Two flaws remained:

1. The engine's /query_cache_hit new_tokens counts shared-L3 hits. Mooncake
   pages are metadata-visible (Exist) before the put finishes, so fast-P
   claimed long prefixes it could not actually serve: the gateway classified
   'small prefill, fast-P has it' while the engine prefetch found nothing
   and recomputed 76k cold — and the probe loop broke at fast-P before ever
   asking the home regular-P, which held an immediate L2 copy.

2. The probe loop stopped at the first answering node, so 'which node holds
   the prefix' was only ever 'fast-P or whoever got asked first'.

Now every fresh candidate is probed (fast pool first, early-out only on a
fully-local prompt), and the routing decision uses local_new = total - l1 -
l2. Ties keep probe order so fresh short prompts with no hits still land on
fast-P.

Co-authored-by: Kimi
2026-10-12 08:17:52 +08:00
minke.yuandKimi 5d38fc96a9 gateway g9i: route small prefills to the node that actually holds the prefix
classify_prefill remembered only new_tokens from the /query_cache_hit probe
and sent every small-prefill request to the fast-P pool. A multi-turn session
whose home P is a regular node therefore landed on fast-P with a local miss:
either a full prefix recompute or a wait for the cross-node L3 backup to
become visible (seconds for long prefixes; observed 10-20s for 76k).

Now the probe records which worker answered. When the prefix lives on a
regular-P the request is Override-routed home (immediate L2 hit), subject to
the same regular_p_queue est-wait 429 gate. Fast-P keeps serving sessions
whose prefix it already holds (its own L2) and brand-new short prefills.

Co-authored-by: Kimi
2026-10-12 08:01:10 +08:00
Liangsheng Yin e6072935bf [DSpark] Apply SGLANG_SIMULATE_ACC_LEN inside the accept instead of switching paths (#43623) 2026-10-11 15:33:24 -07:00
Liangsheng Yin a65b776963 [DSpark] Add SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot (#43670) 2026-10-11 15:18:59 -07:00
Liangsheng Yin 3cc23b3b86 [DSpark] Merge the draft TP sync sites into dspark-draft (#43669) 2026-10-11 15:17:47 -07:00
Liangsheng Yin 00ba90fd1a [DSpark] Run the same markov block in the draft graph tail; drop SGLANG_DSPARK_FOLDED_SAMPLING (#43668) 2026-10-11 15:15:59 -07:00
Liangsheng Yin 48abbdb845 [DSpark] Draft the eager markov block through MarkovSampleStep (#43622) 2026-10-11 15:13:02 -07:00
Liangsheng Yin 467209e838 [DSpark] Add MarkovSampleStep, one op for a markov draft step's add and token pick (#43621) 2026-10-11 15:10:58 -07:00
Liangsheng Yin 9943cf8eef [DSpark] Run every markov head through one step interface (#43620) 2026-10-11 15:07:51 -07:00
minke.yu 261ebbc55b G9i: regular-P queue admission gate (est wait > 10s -> 429 regular_p_queue) 2026-10-12 00:25:52 +08:00
Banruo LiuandShangming Cai d62b9c2480 [KV Events] Publish LoRA adapter names and namespaced block hashes (#42455)
Signed-off-by: Banruo Liu <lichangxinlivingshade@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-11 23:23:30 +08:00
97bfa5cc53 [diffusion] fix: select VSA-H3 for FastH3 when no transformer backend is given (#43690)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:05:25 +08:00
Ke Bao 46a97e5540 Preserve serialized page dtype in NIXL copy fallback (#43593) 2026-10-11 22:39:14 +08:00
Ke Bao b0a75c8196 Share MLA host capacity initialization (#43595) 2026-10-11 22:36:29 +08:00
Ke Bao c938b6e454 Build Mamba host pools from device buffer infos (#43596) 2026-10-11 22:35:52 +08:00
minke.yu 53fb9be7ee G9i: TTFT p95/p99 statistics (observability only, no admission gate) 2026-10-11 22:31:30 +08:00
Shuwen Wang ef3e5c83c5 [HiCache] Keep storage workers alive on backend exceptions (#43550) 2026-10-11 22:26:56 +08:00
minke.yu e5be84f795 G9i: probe classification across pool candidates (fast-P then regular)
t15 stress through the gateway showed /query_cache_hit failing on the
sole fast-P under exactly the load where classification matters (p1
queue 62 / pending 4.7M tokens, 160 probe errors -> fallback). Probe
order is now fast-P least-wait first, regular pool as backup; first
successful answer classifies; unclassifiable (multimodal 400) still
short-circuits; total failure falls back to default routing.
2026-10-11 21:15:59 +08:00
MickandMick Qian 11bfd868ac [diffusion] perf: encode LTX video/audio through the shared bounded MP4 path (#43656)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 21:15:49 +08:00
Shuwen Wang 0923bcd258 [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking (#43567) 2026-10-11 21:12:51 +08:00
FeiHong Guo cf88ed4352 [BugFix][DP] Prevent health checks from perturbing user request routing (#35721)
Signed-off-by: Phoenix3334 <Phoenix3334@users.noreply.github.com>
2026-10-11 21:08:55 +08:00
Shuwen Wang d26b6a5a9c fix: default Inkling to fa4 on Hopper (#35292) 2026-10-11 20:57:42 +08:00
trebladevandMick Qian 586d98cc15 [Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5 (#43062)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 20:55:59 +08:00
minke.yu 39ceba8a3d G9i: router-side pool admission + fast-P routing (gateway)
Pool-aware admission control and small-prefill routing, entirely in
sgl-model-gateway (zero engine changes). Consumes G9 P0 engine surfaces:
/v1/loads (per-dp-rank snapshots, summed to per-node instance-level
views) and /query_cache_hit (new-token classification).

Decision tree per request, before PD dispatch and outside the retry
loop (own 429s are never retried):
  1. D-pool gate: sum of fresh num_running_reqs over decode pool >
     threshold -> 429 d_pool_saturated (fail-open on missing snapshots;
     engine max-running-requests is the hard backstop).
  2. Chat routes: probe least-wait fast-P /query_cache_hit; new_tokens
     < max_new_tokens (tentative 4096) -> fast-P pool.
  3. Pool selection: least estimated wait (queued_total_input_tokens /
     max(drain_rate, floor)) — doubles as the load balancer for the
     multi-fast-P / multi-D future; pool membership is config-only.
  4. Fast-P queue gate: least-wait estimate > threshold (tentative 5s)
     -> 429 fast_p_queue with x-estimated-wait-s.

CLI: --g9i-enabled/--g9i-fast-prefill-url/--g9i-max-new-tokens/
--g9i-fast-p-queue-reject-secs/--g9i-decode-max-running/... all numbers
tentative and centralized (user directive: fixed by measurement later).
Metrics: smg_g9i_decisions_total/rejects_total/no_fresh_snapshot_total,
per-node running gauge, probe latency histogram.
2026-10-11 19:14:40 +08:00
MickandMick Qian 04f740580c [VLM] fix: repair multimodal VMM lease cleanup and source-stream ordering (#43475)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-11 19:13:39 +08:00
efd14cfb66 [DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel (#43070)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yuan Luo <4908075+yuan-luo@users.noreply.github.com>
2026-10-11 19:03:19 +08:00
minke.yu b9d259441c deploy: l3test stack image -> e7dc175e9 (G9 P0 fixes baked in) 2026-10-11 17:47:24 +08:00
minke.yu e7dc175e94 G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
build-sglang-image / build (push) Failing after 21m51s
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
  pool_transfers (comp prepare_prefetch + build_hicache_transfers +
  sidecars) so DeepSeek V4's logical-anchor KV goes through
  batch_exists_v2 side-pool existence instead of v1 batch_exists which
  always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
  backend is attached (not only decode-hicache mode; SWA models forbid
  decode radix+hicache so P nodes could never query L3); construct
  RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
  disagg-prefill result path (it bypasses batch_result_processor, so
  drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.
2026-10-11 17:37:21 +08:00
Sage 53295c6972 [kvevents] fix missing topics in replayed kv events (#43386)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-11 17:25:43 +08:00
Liangsheng Yin 0a21a3b798 [Scheduler] Keep a DP-converted decode batch as the running batch (#43283) 2026-10-11 01:23:51 -07:00
jiaqiang-dot-liu 41652225d0 [Sampler] Sort only the max(top_k) leading columns in the torch sampler (#43346) 2026-10-11 01:00:59 -07:00
minke.yu 6612af9e0e deploy: 归档 b300 实况——l3dummy 镜像升 baba18bc7d + D max-running-requests 512 2026-10-11 15:40:04 +08:00
minke.yu 71c6d8bc55 G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。
2026-10-11 15:38:35 +08:00
Liangsheng Yin dab108b9f3 [Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool (#43290) 2026-10-11 00:18:27 -07:00
Liangsheng Yin 89e24e7a20 [KV canary] Expect sweep counters on the target runner only (#43286) 2026-10-10 23:52:17 -07:00
Liangsheng Yin ad8b1e1f85 [Test] Do not hang scripted-runtime teardown when the server died (#43285) 2026-10-10 23:13:56 -07:00
Liangsheng Yin 448759b235 [Scheduler] Requeue preempted requests when admission admits nothing (#43298) 2026-10-10 22:46:55 -07:00
Qiaolin Yu 047405f534 [qwen 3.8 next] Fuse QSA prefill KV gathering and dtype conversion (#41933) 2026-10-10 22:40:23 -07:00
paulzhang-tm 5678b71b6c [Spec] Allow Qwen3.5 backends to opt into packed target verification (#43037) 2026-10-10 22:39:17 -07:00
Nikolai 22d63f5e43 [AMD]Qwen add Sparse attention to capturing hip graphs (#43383) 2026-10-10 22:03:54 -07:00
Jzz1943 260551ebe9 [KDA] Support Helion MTP verification and harden prefill, decode (#42857) 2026-10-11 12:47:57 +08:00
SunskyXHandXiaoyu Zhang 65e6bb6be7 [GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout (#43332)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 12:43:16 +08:00
a1500b10e3 [diffusion] Don't report video jobs completed after their outputs are deleted (#43084)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 12:42:38 +08:00
Chunan ZengandClaude Opus 5.5 aa5c969dc3 [Perf] Fuse BF16 writes to native HND KV cache storage (#43237)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:41:18 +08:00
Shuwen Wang 58cb1a2be6 [Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks (#43479) 2026-10-10 21:18:41 -07:00
Avaya Aggarwalandniehen6174 3458c94fad [diffusion] comfyui: consolidate executor conditioning-cache fixes (#43492)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-11 11:42:00 +08:00
Avaya Aggarwalandniehen6174 638330d975 [diffusion] comfyui: consolidate FluxAdapter correctness fixes (#43490)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-11 11:41:36 +08:00
Khoa Pham 6576e50cd5 [1/N][DSv41] Enable DP attention with MegaMoE (#43228) 2026-10-10 20:36:35 -07:00
b931b2c8bd [diffusion] perf: run the attention section eager under torch.compile for MiniMax-H3 (#43634)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:02:54 +08:00
b21826e609 [diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths (#43633)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:55:01 +08:00
b237afa547 [diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad (#43632)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:54:18 +08:00
b3a65efe7e [diffusion] perf: keep Cosmos3's fused kernels in the graph (#43577)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:53:43 +08:00
d45ef0f2d2 [diffusion] perf: keep Qwen-Image-2.1 and Ulysses graphs whole (#43586)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:52:43 +08:00
Shangming CaiandKevin Flansburg e1f43c9a61 [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll (#43562)
Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com>
2026-10-11 10:12:34 +08:00
Richard Wang d60e0c8570 [diffusion] Support LLaDA-Image BF16 generation and editing (#37907) 2026-10-11 02:10:51 +00:00
e4b423256e [diffusion] fix: serve whole graphs after offload-during-compile (#43631)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:03:47 +08:00
Richard Wang d7ed5b56f5 [diffusion] fix: keep the overlay probe from stalling cached startup (#43590) 2026-10-11 10:01:01 +08:00
156d54e56c fix: wait for HiCache load-back before hybrid MLA KV reads (#38217)
Co-authored-by: zhouzechun <zhouzechun@kuaishou.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-10-11 09:50:13 +08:00
Jzz1943andXiaoyu Zhang 9336681310 [DSA] Support TileLang FP8 KV cache on CUDA (#42957)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-11 09:12:18 +08:00
Xiaoyu Zhang 33d2ef1756 [Fix] Reserve remaining tokens for ignore_eos requests during prefill admission (#43564) 2026-10-11 09:05:48 +08:00
WenhaoZhangandOnePunchMonk 5730345717 [diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode (#43495)
Co-authored-by: OnePunchMonk <aggarwal.avaya@yahoo.com>
2026-10-11 08:58:23 +08:00
d1313dd5fe [diffusion] fix: skip inductor's saved multi-kernel pick for torch.compile (Z-Image warm starts crash or hang) (#43601)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:56:55 +08:00
3022989bad [diffusion] CI: skip the LTX-2 RMSNorm-modulation guard test off the CUDA platform (#43574)
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:54:37 +08:00
65386cb95a [diffusion] perf: add the dynamic LoRA delta inside the second GEMM (#43501)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:50:53 +08:00
e328c7ecbd [diffusion] fix: skip QKV packing while layers are offloaded (#43576)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:47:35 +08:00
7373665085 [diffusion] fix: keep exported buffers out of expandable segments for IPC all-to-all(#43584)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:46:30 +08:00
5e872557cb [diffusion] feat: keep the TP all-reduce inside the graph (#43602)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:45:54 +08:00
664e4c6499 [diffusion] feat: warn instead of disabling on unvalidated models for BCG(#43568)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:40:28 +08:00
23764b1242 [diffusion] feat: bind each GPU worker to its GPU's NUMA node (#43565)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:38:58 +08:00
9933cf5437 [diffusion] feat: declare regional compile regions for flux (#43575)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:33:29 +08:00
68e1a3dbb1 [diffusion] fix: accept request-level profiling on the image API (#43573)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 08:30:20 +08:00
d32d8fa0d1 [Dsv4.1] Mega-mhc integration (#42808)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-10 17:25:17 -07:00
Liangsheng Yin 7187bb593e [Model] Keep gpt-oss router logits batch-invariant under deterministic inference (#43624) 2026-10-10 17:22:29 -07:00
Mohammad Miadh AngkadandMohammad Angkad c640dc97ce [CI] Fix Kimi-Linear DCP PD test startup (#43598)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-10 16:53:54 -07:00
Cheng Wan 234ccba7f6 [Refactor] Declare every model's stages from a shared function and drop the meta-device neighbour build (#43460) 2026-10-10 12:43:09 -07:00
Cheng Wan 3b16eeedce [Refactor] Declare pipeline neighbours from a shared declaration function (#43459) 2026-10-10 12:42:46 -07:00
Cheng Wan 3b717c0f13 [Refactor] Decide an exit's fixed facts when the stage is built (#43458) 2026-10-10 12:42:23 -07:00
Cheng Wan a956aa192a [Refactor] Bind a layer stack in one pass over each stage's data flow (#43457) 2026-10-10 12:42:00 -07:00
Cheng Wan d2c597c8db [Refactor] Decide input-scattered eligibility and stage rejections in one place (#43456) 2026-10-10 12:41:38 -07:00
Cheng Wan b0dde87de2 [Refactor] Make residual read and update capabilities explicit and checked (#43455) 2026-10-10 12:41:16 -07:00
Cheng Wan 514feb1afc [Fix] Keep residual rows and sums consistent across appends and pipeline handoffs (#43454) 2026-10-10 12:40:53 -07:00
Cheng Wan 8bd7f2d178 [Refactor] Record what a bound stage path does instead of reading it back (#43453) 2026-10-10 12:40:30 -07:00
0acd93c35c [Scheduler] Use committed length in spec mamba track boundary check (#43029)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-10 12:29:40 -07:00
Shuwen WangandKe Bao f9cee8d2b2 [HiCache] Keep protected cgroup page cache charged in host budgets (#42788)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-10-10 17:03:40 +00:00
Xiaoyu Zhang f61163b4b0 [Docs] Let the Qwen3 B300 recipe use the default attention backend (#43554) 2026-10-10 10:00:04 -07:00
Xiaoyu Zhang 587b6d622d [Docs] Document Qwen3-4B/8B H200 mixed-chunk latency tradeoffs (#43551) 2026-10-10 09:59:46 -07:00
minke.yu dbc661602e deploy: 默认开 SGLANG_HICACHE_PREFETCH_IO_THREADS=4(io_aux 4 lane,mc4/mc8 +23%)
- 11 个挂 L3 的 compose 模板 + k8s 3p1d RBG manifest 的 prefill-base 统一显式落 4
  (代码默认已是 4,显式化防漂移;同一实例所有 rank 必须一致——crc32(rid)%N 分发
  + 每 lane 独立 gloo 组,不一致会集合通信错位挂死)
- D 角色不挂 L3,无需加
- A/B 数据见 investigations/l3-hicache-deep-dive-20261010.md §8
2026-10-10 23:07:45 +08:00
Shunkangz de487f8039 [kv-shard 3/4] Enable Control Plane C (#40102) 2026-10-10 22:53:58 +08:00
huangtingwei 406540ab5f [HiCache] Add staged write-back for the page-unified KV cache layout (#39606) 2026-10-10 22:49:45 +08:00
Shuwen Wang 99d6f4e158 [MemCache] refactor: require an anchored lock receipt for non-root releases (#41452) 2026-10-10 22:43:49 +08:00
Shuwen Wang 44bc478863 [HiSparse] Avoid host synchronization in eager backup (#41446) 2026-10-10 21:49:55 +08:00
minke.yu 054dcee896 tier compose:跨实例 P2P 的 extra_hosts 互配(A/B 互加对方 rank 主机名别名,否则副本跨端分配后 open segment DNS 失败) 2026-10-10 21:45:16 +08:00
hhy-seven 616178a596 [Responses] Fix max_output_tokens overestimate on multimodal requests (#35524) 2026-10-10 13:42:53 +00:00
a510340aeb [diffusion] perf: run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout (#43508)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:34:54 +08:00
72657f7ca5 [diffusion] perf: pipeline every USPAttention Ulysses path's exchange over the copy engine (#43521)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:34:18 +08:00
minke.yu 73a25267ac tier realstore-x2 compose:第二实例(GPU 2,3 / 端口 30041 / localhost-b 前缀)做跨 P 实例 L3 读写验证 2026-10-10 21:28:21 +08:00
Shuwen Wang 416ebe6090 [CI] Fix EAGLE3 CUDA parity prefill chunk alignment (#43538) 2026-10-10 21:11:17 +08:00
49e629d239 [Rust] Upgrade sglang-radix-tree to PyO3 0.29 (#42807)
Co-authored-by: colleen-tml <296620403+colleen-tml@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:56:50 +08:00
minke.yu bcad628d30 tier realstore compose: 显式 SGLANG_HICACHE_PREFETCH_IO_THREADS=4(io_aux lane 数,配合 baba18bc7d) 2026-10-10 20:05:54 +08:00
minke.yu baba18bc7d hicache L3 prefetch: io_aux 传输并行化(按 op 分 lane)
build-sglang-image / build (push) Failing after 48m34s
打点实测单条 prefetch_io_aux_func 线程串行传输是所有请求的瓶颈:
mc8 下每请求 xfer 排队 0->1.9s 线性增长(每请求 5 批 x ~68ms 固定开销,
介质无关)。本提交把传输层拆成 N 条并行 lane(默认 4,
SGLANG_HICACHE_PREFETCH_IO_THREADS 可调):

- dispatch_prefetch() 用 crc32(request_id) % N 分 lane(内置 hash 对 str
  每进程加盐,跨 rank 不一致会导致 gloo 集合通信错位,禁用)
- 同一 op 的所有 chunk 保持在同一 lane,保序 reduce 语义不变
- 每 lane 独立 io 队列 + io_aux 线程 + sync 队列 + sync 线程 +
  独立 gloo completion 组(集合通信按 lane 过滤后的分发顺序保序)
- query 阶段的 hit_reduce 仍在单 prefetch_thread,不动
- 生产点(hiradix/unified/hybrid pp-ticket)改调 dispatch_prefetch
- 模拟器与两个单测文件同步适配

注意:SGLANG_HICACHE_PREFETCH_IO_THREADS 必须在实例内所有
TP/CP/PP rank 保持一致,否则 lane 组上的集合通信会错位挂死。
2026-10-10 19:04:49 +08:00
Bingxu Chen a916d91cd3 [AMD][CI] Cap pyarrow below 26 for lmms-eval (#43547) 2026-10-10 18:59:49 +08:00
minke.yu 0afcb334a4 realstore 回退 STORAGE_BATCH_SIZE=512:TCP 传输下大批次倒退(mc8 668k→320k),dummy shm 才受益;真实 store 保持默认 128 2026-10-10 18:16:56 +08:00
minke.yu 33e6381b7d tier dummy compose 对齐 b300 实况:wait_complete + HC_TIMING + STORAGE_BATCH_SIZE=512 2026-10-10 18:06:48 +08:00
minke.yu 9c4ff32562 tier compose:STORAGE_BATCH_SIZE=512(摊薄每批 ~60ms 固定开销,dummy 实测 mc8 +26%) 2026-10-10 18:05:16 +08:00
minke.yu 7d5759b2d9 hicache: STORAGE_BATCH_SIZE 可env化(SGLANG_HICACHE_STORAGE_BATCH_SIZE,默认128不变)
build-sglang-image / build (push) Failing after 18m40s
打点显示每批存储 IO 有 ~30-70ms 固定开销(与传输介质无关),
调大批次可摊薄。env 化便于不动代码做 128/512/1024 对照。
2026-10-10 17:55:14 +08:00
Sage 6fc8d9da32 [rust-server] preserve PD routing in chat and completions (#42989)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-10 17:29:46 +08:00
minke.yu ac11e9162d tier realstore: extra_hosts 别名让 rank 后缀主机名可解析
mooncake P2P 模式下段名/te_endpoint 内嵌 local_hostname 且需 DNS 可解析;
rank 后缀名(localhost-d0t0p0c0 等)必须配 /etc/hosts 别名指向 127.0.0.1,
否则跨段 openSegment 握手失败(TRANSFER_FAIL 全灭)。
验证:mc8 402k→668k tok/s,读路径约 40% 走 memcpy 60% 仍 TCP。
2026-10-10 17:14:18 +08:00
Liangsheng Yin c55790d38b [CI] Drop CPU tests with no bug-catching signal and run Rust workspace tests in their own workflow (#43503) 2026-10-10 02:13:53 -07:00
minke.yu 84fc632088 mooncake_store: local_hostname 加 rank 后缀,让 master 的 preferred-segment 分配落到本进程段
build-sglang-image / build (push) Failing after 1s
两个 rank 进程共用默认 localhost 时,master 按名字匹配无法区分两个段,
副本会落到对方进程段;mooncake 的 local memcpy 快路径要求同进程 endpoint
(跨进程虚拟地址无效),于是静默回退 loopback TCP(实测 ~0.78GB/s)。
唯一化后自写自读可命中 memcpy(干净集群探针实测 lo 流量归零)。
注意:memcpy 线程池 kDefaultMemcpyWorkers=1(mooncake 0.3.13 硬编码),
单流 ~1GB/s,进一步提升需改 mooncake 并重编译。
2026-10-10 16:42:00 +08:00
siliangchen-amd ff19159e16 [AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears (#41071)
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com
2026-10-10 01:31:20 -07:00
minke.yu 8d9bce30a7 tier realstore: 开 HC_TIMING 打点 + prefetch policy 改 wait_complete
best_effort 下低并发请求 compute 赢比赛、L3 命中不生效(mc1/2/4 实为重算);
wait_complete 后真实 L3 命中:mc1 TTFT 2989→661ms,mc8 聚合 203k→402k tok/s。
2026-10-10 15:54:11 +08:00
chuyeh e35f60b9bc [AMD][CI] Drop removed tp_rank/tp_size kwargs from small-M FP8 proj test (#43318) 2026-10-10 00:52:28 -07:00
d7d22e8a30 [AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size (#39840)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-10-10 00:47:13 -07:00
minke.yu 5f2ffa70b8 hicache: SGLANG_HICACHE_TIMING 打点 L3 预取全链路分阶段耗时
build-sglang-image / build (push) Failing after 1s
env 开关(默认关),开后在以下阶段输出 HC_TIMING 日志行:
query_wait/query(含 batch_exists 逐批)/hit_count all_reduce/
xfer_wait/xfer(含 batch_get 逐批)/ack_reduce/ack_drain 排队/
graft(insert_host)/prefetch_e2e/loadback。
目的:定位 L3 命中单请求 ~3s(≈重算速度)的时间分布。
真实 mooncake store 与 dummy store 实测吞吐一致,排除后端实现因素。
2026-10-10 15:29:01 +08:00
d8c5eca3ac [diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away (#43385)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:02:03 +08:00
Richard Wang 441de659f4 [CI] Fix Python 3.10 failure in prefill complete integration test (#43505) 2026-10-10 00:00:03 -07:00
tao 6e8f3e09dd [Model Loader] Add InstantTensor loader support (#40453) 2026-10-09 23:01:59 -07:00
minke.yu f4ef648b16 tier L3 A/B:dummy→真实 mooncake store(线上同构 master+P2PHANDSHAKE+tcp)+ 零预热 mc 扫描脚本
dummy 后端实测单请求 L3 命中 2.96s/54k tok/s(零预热修正后),聚合 ~200k;
CPU 在 scheduler native 线程、store 守护进程与 Python 工作线程均空闲,
疑似 dummy 后端 batch_get 内部 per-key 串行。切真实 store 复核以区分
sglang L3 路径问题 vs dummy 调试后端工件。注意 bench 必须 --warmup-requests 0,
默认 warmup 会把 prompt#0 种进 L1/L2 污染 L3 测量。
2026-10-10 13:59:31 +08:00
cctry 33f6b00f32 Clamp default decode CUDA graph max_bs to max_running_requests (#43172) 2026-10-09 22:54:32 -07:00
Mohammad Miadh AngkadandMohammad Angkad 16be9df25f [CI] Remove the GLM-5.3-Flash PTX KDA prefill nightly test (#43470)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 22:52:17 -07:00
Yuwei An 8b2aaf7060 [Refactor][TCPCG] Retire backend selection and configuration (3/9) (#42287) 2026-10-09 22:42:35 -07:00
Yuwei An 0aa580f904 [Dsv4.1] Add DeepSeek V4.1 DSPARK TP and DP EP accuracy coverage (#43478) 2026-10-09 22:41:30 -07:00
YAMY 17852cdfe8 [DSv4.1] Initialize the full-to-SWA mapping for request-window pools (#43165) 2026-10-09 22:39:26 -07:00
Frederik Gossen 3831e7e091 [diffusion] Z-Image: autotune the shared transformer block at both token scales (#35755) 2026-10-10 13:04:43 +08:00
Raymond HuangandClaude Opus 5 aa79087017 [diffusion] fix: fail MiniMax-H3 quality="high" closed on substituted DiT weights (#34978)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-10 13:03:44 +08:00
ccae3b3639 [diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell (#43373)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:58:30 +08:00
Shangming Cai 1814ece57f [PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages (#43329) 2026-10-10 12:38:38 +08:00
cctry 20948cdcc6 [PD] Reclaim parked optimistic prefills when request slots are exhausted (#43442) 2026-10-09 20:31:24 -07:00
Liangsheng Yin 7d44269907 [MemCache] Fix the strict busy check on prefill-aware SWA and tidy small leftovers (#43474) 2026-10-09 20:27:27 -07:00
f2e9a2bed1 [unified memory pool] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts (#43206)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-09 20:19:30 -07:00
MickandMick Qian a65f4ffba5 [diffusion] docs: standardize diffusion cookbook command builders (#43472)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 11:14:31 +08:00
436d73ddda [diffusion] model: support HunyuanImage-3.0-Instruct (#36311)
Co-authored-by: Elizaveta Martirosian <you@example.com>
Co-authored-by: wes-lyu <294033513+wes-lyu@users.noreply.github.com>
Co-authored-by: Wes <its.wesley.lyu@gmail.com>
Co-authored-by: AuFlow <73925903+AuFlow@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-10 11:11:28 +08:00
506f140166 [sgl-router] Read finished /generate streams to the end; expose reorg admission caps as flags (#43233)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 10:59:09 +08:00
minke.yu 6469cf4197 deploy: tier bench scripts (cp2 L1/L2/L3 matrix + R7 async aggregation) 2026-10-10 10:50:24 +08:00
Benjamin TruongandXiaoyu Zhang 37678784c6 [Gemma4][Attention] Enable Gemma4 FA4 on SM90 with head-dim 512 support (#42019)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-10 10:40:19 +08:00
55a8dffc87 Skip the eager activation reserve in KV sizing on a capped Full prefill role (#43435)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-09 19:36:48 -07:00
a338b9858a [PD] Receive decode KV into separate paged SWA host pools; let host-capable receivers own abort drain tracking (#43410)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 19:33:06 -07:00
Vladislav NosivskoyandBBuf 2ff3e6dce1 [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 (#42604)
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-10 10:29:42 +08:00
Liangsheng Yin 465253ef52 [Scheduler] Fix the deterministic-inference hang when a prefill chunk is shorter than its alignment (#43444) 2026-10-09 19:06:18 -07:00
78218194ad perf(kda): rebalance MTP verify precompute and state synchronization (#42683)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-10 10:03:19 +08:00
minke.yu 3f8751c6ed 新增 tier-cp2 单机 compose:cp2+dspark+L2 6.4M+L3 shm dummy,测 计算/L1/L2/L3 四档 prefill 极限与异步聚合 2026-10-10 09:51:09 +08:00
Michael 37114afe20 [AMD][Fix] Saturate FP8 KV commits instead of reproducing e4m3fnuz NaN overflow (#42982) 2026-10-10 09:49:56 +08:00
MickandMick Qian 77ba54f7df [diffusion] docs: add Wan-Animate-2 cookbook command picker (#43380)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:45:16 +08:00
16423cb6d2 [sgl-router] Serve DeepSeek-V4.1 Flash through sglang-processor (#43000)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:23:09 +08:00
5f94a45896 [srt] Allow mixed chunk prefill on the Full prefill CUDA graph backend (#43461)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-09 18:21:22 -07:00
minke.yu e4f1310d6f l3dummy: hicache-ratio 1.0→4.0(L2=3.2M token/P)——#38348 bench 显示收益被 L2 容量掩盖(host_capacity 拒绝 0.75M~2.97M token/P),放大 L2 验证热轮命中不被容量拒、L3 hit 嫁接后在 L2 留存 2026-10-10 09:14:07 +08:00
MickandMick Qian cde16aa64d [diffusion] model: support Qwen-Image 2.1 Turbo and checkpoint sampling grids (#43391)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:12:56 +08:00
e8fb52c526 [Fix] Keep request-pool row generations monotonic across clear() (#43439)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: colinchan15 <colinchan15@users.noreply.github.com>
2026-10-09 18:08:54 -07:00
ashwini rathi 8cf4d73c84 [XPU] Fix XPU CI breaks from #41105 (KDA chunk_offsets) and #42588 (int4 test tp_group) (#43334) 2026-10-10 09:06:24 +08:00
Avaya Aggarwal 66e6b68d82 [diffusion] comfyui: report the real runtime import error (#43178) 2026-10-10 09:05:12 +08:00
ff3aa58f11 [diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per war while bit-identical (#43337)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:03:36 +08:00
bookinstockandMick Qian 32f34e898b [diffusion] fix: honor requested inference step count for MiniMax H3(#38671)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-10 09:02:57 +08:00
bd7c7971ba [diffusion] perf: move MiniMax-H3's pipelined Ulysses copies off the SMs (#43361)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:58:07 +08:00
Alison Shao 9b37e28b83 Disable DualGemm fusion for Inkling dense MLP (#43447) 2026-10-09 17:56:20 -07:00
6acaf3374a [kernel] SwiGLU-OAI MXFP8 producer; move swizzled MXFP8 producers to ops/quantization (#42810)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Denji-kk <51020465+Denji-kk@users.noreply.github.com>
2026-10-10 08:51:57 +08:00
Rain Jiang 961bcf481d support DeepEP v2 on GLM-5.3 (#43432) 2026-10-09 16:57:12 -07:00
minke.yu a42ab6fce0 测试适配 #38348 新语义:SWA 双层缺失的 backuped 节点现在是合法匹配边界
build-sglang-image / build (push) Failing after 1s
旧语义(host_hit=0 + branching=9 + 重插建窗)被 fail-soft 取代:
FULL 段直接 host 命中(host_hit=leaf_len、branching=None),load-back
落 FULL 后由续写请求的 insert overlap 从 Full KV 重建窗口 SWA。
重写 test_swa_branching_seqlen_uses_host_full_hit 验证新链路:
match 命中 → load-back → 插入扩展 token 触发 overlap 重建 → rematch
设备全命中且洞口节点的窗口 SWA 已恢复。
2026-10-10 07:42:44 +08:00
Yuwei AnandYuwei An 2d38934493 [DSv4.1] mxfp8 dispatch cache (#42156)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
2026-10-09 16:36:28 -07:00
minke.yu fa8ad18b5c 手工移植上游 #38348:hicache SWA prefetch fail-soft 降级
build-sglang-image / build (push) Failing after 1s
SWA 池 L3 取回短缺时不再整批丢弃 prefetch 结果:保留已取回的 Full KV
前缀,仅丢弃失败的 SWA 池(释放其 host buffer、从 comp_xfers 摘除),
load-back 时经既有 SWARebuild/RecoverSWAWithLockedFull 路径从 Full KV
重建窗口。Mamba 等不可重建池短缺时仍走原 all-or-nothing 整批丢弃。

配套改动(同 PR):
- swa.py validator:SWA 双层缺失但节点已 backuped(Full KV 在 host)时
  放行作为匹配边界;LOAD_BACK 收集遇双层缺失空洞时 break 而非 assert。
- prefetch_from_storage:用完整匹配前缀重算 last_hash,修复 anchor 节点
  无 host_value(异步 backup 未完成/降级嫁接)时哈希链锚错位。
- 适配差异:本树有 sidecar_hash_values 重构,belief 自愈沿用
  invalidation_hashes + trimmed_pages 语义;ongoing_prefetch 以
  CacheRequestHandle 为键,comp_xfers 改为显式参数传入检查函数。
- 单测同步更新为 fail-soft 语义(TP 一致性降级、失败池释放无泄漏、
  LOAD_BACK 锚定空洞停止收集)。
2026-10-10 07:23:15 +08:00
Liangsheng Yin 46498bc8c5 [MLX] Sync decode KV to the pool once at release, clamped to the owned prefix (#43292) 2026-10-09 16:20:45 -07:00
Liangsheng Yin 114cb44920 [MLX] Read radix prefix slots from the model runner's req_to_token pool (#43291) 2026-10-09 16:18:58 -07:00
Liangsheng Yin 8020ea2ee9 [MLX] Drop a retracted request's runner state at its re-prefill (#43284) 2026-10-09 16:16:59 -07:00
Liangsheng Yin f7e601b936 [MLX] Drain in-flight jobs before a retract-mode pause_generation (#43282) 2026-10-09 16:14:23 -07:00
f6c848c882 [PD] Use 1 to disable decode host receive and 0 for always-on (#43407)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 15:50:52 -07:00
Alison Shao f6e3444b2e ci: diffusion-only option for the runner utilization report (#41620) 2026-10-09 15:47:47 -07:00
Cheng Wan 58f6976ce7 [Fix] Unblock CUDA graph startup for hybrid parallelism (#43211) 2026-10-09 15:39:59 -07:00
cb9bd1d4d4 [Perf] Reduce multi-detokenizer router IPC sends (#42238)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-09 15:32:14 -07:00
66d840a8ef [cuda_graph] Size dp-local decode graphs by the runner's own gather requirement (#43303)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-09 15:21:23 -07:00
Kevin MiandClaude Opus 5.5 6d8ddaf1d7 [AMD] MiniMax-M3 indexer CP: packed scoring for EAGLE verify rows (#42614)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:05:50 -07:00
fbf8d730fd [Fix] MiniMax-M3: default to the breakable prefill CUDA graph (#41845)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:05:05 -07:00
Kevin MiandClaude Opus 5.5 70e82f7b73 [AMD] Run the MiniMax indexer CP test on MI35X, not MI300 (#43259)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:04:25 -07:00
weireweireandweireweire 46c2585faa [DSpark] Fold draft sampling for DeepSeek-V4.1 heads under AUTO (#42987)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-10-09 14:40:28 -07:00
Huabin 2f1acef3d8 [Model] Add LLaDA2.2 Block Routing MoE support (#31768) 2026-10-09 12:14:44 -07:00
Mohammad Miadh AngkadandMohammad Angkad 72eab59bf6 [GLM-5.3-Flash] Enable breakable prefill CUDA graph by default (#42845)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-10 02:41:02 +08:00
87845cda2b [Multimodal] Chunk and cache audio items whose offsets span several placeholder runs (#43301)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com>
2026-10-09 10:57:01 -07:00
3bffe69b74 [moe] Fill num_token_non_padded for the customized A2A backend at EP 1 (#43245)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: cctry <cctry@users.noreply.github.com>
2026-10-09 10:55:02 -07:00
3c612132fd Overlap scheduler startup with the data parallel controller (#43177)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
2026-10-09 10:54:22 -07:00
minke.yu 7f1edbba80 cherry-pick 上游 #40429:FlashMLA sched meta 超 48KiB smem 时跳过预计算(encoder/decoder replay 的大 batch replay pass 会触发 smem 溢出崩溃)
build-sglang-image / build (push) Failing after 1s
2026-10-10 01:20:15 +08:00
minke.yu 99c317b131 cherry-pick 上游 #42716(运行时部分):encoder SWA bounded replay 下放行 host 层前缀命中——request_window 形态的 SWA 不进树,校验门不应拦截 host-only 节点;rust adapter/test 两处 hunk 与本树结构不符未合(默认 python 树芯用不到)
build-sglang-image / build (push) Failing after 1s
2026-10-10 01:02:33 +08:00
minke.yu 2b2ede8211 l3dummy 定型为 shm+best_effort(唯一正收益组合 +26.4%);wait_complete 因 discard 白等 -17.7% 弃用 2026-10-10 00:09:02 +08:00
minke.yu b4dbcebbec mooncake standalone(dummy) 模式打通 dsv41 unified cache:放宽 standalone 检查对 LogicalHostPool 锚点的误拦(实体 buffer 由旁池运行时 register_buffer);新增 l3dummy compose(单 store 服务 64GB shm 段 + fake-cargo 跳过 rustup 下载)。实测 24x160k 热/冷 +26.4%(vs TCP best_effort -4.4%)
build-sglang-image / build (push) Failing after 1h10m35s
2026-10-10 00:06:32 +08:00
Mohammad Miadh AngkadandMohammad Angkad 5cbf949839 Fix torch.compile crash in fused gate-sigmoid-mul launcher (#43384)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 23:51:30 +08:00
76ccf0850c [diffusion] fix: fix TeaCache CFG state lifecycle and skip boundaries (#42422)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-09 23:20:13 +08:00
121aa8dcc7 [diffusion] perf: fuse MiniMax-H3's RMSNorm and indexed AdaLN under quality lossless (#43327)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:39:11 +08:00
Jimmy Shong 408f5d057a docs: add Clef accuracy results to the deployment wizard (#43349) 2026-10-09 22:19:27 +08:00
1203eb9181 [diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against attention over the copy engine (#43164)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:04:29 +08:00
xiaofei-zheng fba3ad4b1f [Frontend] Parallelize long chat prompt encoding (#41259) 2026-10-09 21:30:22 +08:00
xiaofei-zheng db32673b78 [AMD] Let the GLM DSA NextN draft declare its own shared-expert fusion architecture (#41258) 2026-10-09 21:28:45 +08:00
xiaofei-zheng d5c3ae9752 [AMD] Eliminate remaining gfx95 FP8 scale relayout copies (#41030) 2026-10-09 21:27:38 +08:00
78996e3f02 [Performance] Batch Mooncake hybrid-pool metadata and puts (#39696)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:12:42 +08:00
d511e6bd1d [sgl-router] Parse tool calls when serving chat through /generate (#42999)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:07:30 +08:00
67d063f63b [diffusion] feat: allow per-role attention backend override (#35310)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 20:57:29 +08:00
348689da40 [PP+PD] Fix #38206 Reduce bootstrap consensus latency (#38959)
Co-authored-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 20:53:48 +08:00
f1e0472fcc [diffusion] model: support Wan-Animate-2 (#42941)
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 20:46:23 +08:00
minke.yu 30ba5818e7 l3test: prefetch 策略 wait_complete→best_effort(实测等待阻塞净亏 7-17%,best_effort 持平 -4.4%)
build-sglang-image / build (push) Failing after 1s
2026-10-09 20:30:46 +08:00
Shuwen Wang 0955169ed2 [mem_cache] Bound FlexKV checkpoint stores and restore DSPARK cache checks (#35694) 2026-10-09 20:25:33 +08:00
59eb71a831 [diffusion] fix: align SGLD CFG combine formula with diffusers Z-Image (#23772)
Co-authored-by: SGLang CI <ci@sglang.ai>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-09 19:47:03 +08:00
minke.yu c9bfb24bcc l3 模式:默认 48×160k + 冷热轮间 120s 备份排空 2026-10-09 19:31:45 +08:00
minke.yu 3d109d50be fix: l3test L2 改 ratio 1.0(0.05 会噎死 L3 写入路径) 2026-10-09 19:30:53 +08:00
Shuwen Wang f48ed2127e test: allow bounded checkpoint lag in Mamba HiCache tests (#42523) 2026-10-09 19:24:39 +08:00
3805d78796 [sgl-router] Serve /v1/chat/completions through /generate, with reasoning parsing (#42998)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-09 19:17:52 +08:00
minke.yu 4eabf5e228 G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
  记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
  set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费

B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
  + query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
  _tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
  v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit
2026-10-09 18:48:27 +08:00
Xiaoyu Zhang 0eb24e7d5b [Performance] Optimize Qwen3.8-Flash-Next BF16 decode on Blackwell (#42175) 2026-10-09 18:46:12 +08:00
516fd1c1b5 [diffusion] perf: tile MiniMax-H3's indexed modulation kernels by 2048 columns (#43267)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 18:23:52 +08:00
minke.yu 7d67d57804 fix: l3test 变体改用 hicache-ratio 0.05(V4 不支持 hicache-size) 2026-10-09 17:55:02 +08:00
Xinyuan Tong 91b3692a4c docs: add thinking budget guide (#41601) 2026-10-09 17:50:34 +08:00
BingjiaWang 888de24cc8 Add BJWang-ant to CI_Permission (#43321) 2026-10-09 02:31:38 -07:00
黄孝君 e1c6ddca1d [NPU] Bump SGLANG_KERNEL_NPU_TAG to 2026.9.0.post9 (#43340) 2026-10-09 17:30:38 +08:00
vOvandXiaoyu Zhang 7cfecbef56 [diffusion][model] Add native SANA-Video 2.0 support (#41492)
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-09 17:28:03 +08:00
DevashishLal-CB 4debae7534 Add blackwell dual gemm fusion support for MLP (#41887)
Signed-off-by: Devashish Lal <devcode@meta.com>
2026-10-09 17:26:33 +08:00
minke.yu 491989284f deploy: 3p1d-l3test 变体(P 缩 L1/L2 强制 L3 命中) 2026-10-09 17:26:04 +08:00
Thomas Wang 5eca4d8caf [AMD] Set maxseq length limit for DSV4.1 BCG metadata buffers (#42570) 2026-10-09 02:15:43 -07:00
Xinyi Song 23f890613e [AMD] Patch aiter's group32 MXFP8 GEMM masked-scale fix (#43319) 2026-10-09 02:08:44 -07:00
minke.yu 53ea8fb2a7 report: 增加 input tps 列 2026-10-09 17:02:23 +08:00
minke.yu efdbfd2cc3 fix: awk sprintf 括号+TPOT/TTFT 取最后一列 2026-10-09 17:01:35 +08:00
minke.yu 8342b89e85 fix: awk sprintf 括号 2026-10-09 17:01:05 +08:00
Jimmy Shong 14e3414513 [Docs] Add Clef cookbook with selectable Clef Flash variant (#43235) 2026-10-09 01:56:45 -07:00
amote-i ff146fa20c [NPU] [DOC] remove --enable-torch-compile from Ascend support matrix (#43336) 2026-10-09 16:55:39 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7578432c21 [LoRA][Test] Use the 0.9 ROUGE-L tolerance for the CUDA multi-LoRA batch test (#43297)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-09 01:51:01 -07:00
minke.yu c92938d718 deploy: bench_suite_v2 (warmup/prefill冷热双轮/decode阶梯80%cg/L3语料溢出) + 单独P/D基线compose 2026-10-09 16:50:23 +08:00
Avaya AggarwalandMick Qian d677848968 [diffusion] fix: fix CFG normalization to use per-sample norm instead of batch-flattened norm (#41706)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:43:18 +08:00
Murthy LandClaude Opus 5 3ab142adc4 [diffusion] feat: enable breakable CUDA graphs for FLUX.2 Klein (#36470)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-09 16:40:09 +08:00
minke.yu 7578cdd218 revert: #42035 continuous prefill input polling (CP rank collective mismatch deadlock)
build-sglang-image / build (push) Failing after 39m1s
compose 3p1d regression on 1f6ca2cf6: all 3 P nodes wedged with split-brain
CP ranks (one blocked in _sync_hicache_ready_counts all_reduce, partner in
request broadcast). Root cause: #42035 wraps get_next_disagg_prefill_batch_to_run
(which contains the hicache attn_cp_group all_reduce) in rank-local guards
(len(result_queue)<2, _deferred_input_requests, skip_batch_creation); any
cross-rank divergence in batch-None decisions misaligns collective pairing
permanently. Continuous polling requires attn_cp_size==1 anyway, so cp2
deployments get zero benefit. Reverts 93d610c1fd + test adaptations
6ba34507f1/bde1fcbf51/1f6ca2cf68 + deploy bump 236686454c.
Keeps #42194 (9ef65b51c2) and checkpoint_disagg_prefill shim (c4f3e3be33).
Evidence: D:\B300\investigations\pd-deadlock-20261009.md
2026-10-09 16:39:06 +08:00
b922f5ca20 [diffusion] fix: share the IPC all-to-all transport between AllToAll4D and USP (#43269)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:31:55 +08:00
MickandMick Qian 17b2f35ae7 [diffusion] refactor: remove unused helpers and write-only state (#43064)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:30:55 +08:00
James XuandMick Qian ee4e2f7818 [diffusion] optimization: enable channels_last_3d for Cosmos3 VAE (#28021)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:29:46 +08:00
Cheng Wan dc0b394471 [Refactor] Remove Kimi K3's own layer communication and let the consumer set an FFN's exit (#42936) 2026-10-09 01:27:46 -07:00
Cheng Wan 305ad5503d [Refactor] Run Kimi K3's fused collectives through stage boundaries (#42898) 2026-10-09 01:27:23 -07:00
Cheng Wan 2b469040d2 [Refactor] Support attention-TP dense FFNs and unpadded batches on stage boundaries (#42897) 2026-10-09 01:27:00 -07:00
Cheng Wan b9ae3d1a65 [Fix] Run attention on a padded extend batch's real rows (#42896) 2026-10-09 01:26:38 -07:00
Cheng Wan 7203747cc8 [Fix] Shard Kimi K3's shared experts over the TP group they sum over (#42895) 2026-10-09 01:26:15 -07:00
MickandMick Qian 45ad463c87 [diffusion] refactor: deduplicate UniPC solvers and Gemma3 shard loading (#42899)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 16:26:04 +08:00
Cheng Wan 125f88cb75 [Refactor] Run Kimi K3 through stage boundaries (#42894) 2026-10-09 01:25:51 -07:00
Cheng Wan 666da4f56b [Fix] Kimi K3 KDA value head size and SiTU input alignment (#42893) 2026-10-09 01:25:28 -07:00
Cheng Wan 48a6c0a5d0 [Fix] Return an FFN's output to the attention rows at a pipeline handoff (#42892) 2026-10-09 01:25:05 -07:00
Cheng Wan b2402ccad6 [Fix] Merge DeepSeek-V4 two-batch-overlap microbatches without a residual (#42891) 2026-10-09 01:24:42 -07:00
minke.yu 21f7ce37dd build: 镜像内编译 sgl-model-gateway(Rust gateway,G9 前置)
build-sglang-image / build (push) Failing after 31m52s
三个 Dockerfile(b300 全量/fast/gitea)在 COPY 源码后加 cargo 层:
cargo build --release -p sgl-model-gateway -p sgl-model-gateway-python,
RUSTUP_TOOLCHAIN=stable 绕过 rust-toolchain.toml 1.90 钉版(基底自带 1.98.1),
BuildKit cache mount 持久化 registry/target(冷 ~15min,增量秒~分钟级),
python 扩展产物重命名 sglang_router_rs.abi3.so 覆盖 python 包内同文件。
2026-10-09 b300 实测:crates.io 直连无需镜像,冷编译成功。
2026-10-09 16:07:36 +08:00
b130f8af02 [sgl-router] Serve /v1/completions through /generate, with the shared OpenAI layer (#42997)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:04:04 +08:00
f620d733d2 [rust-processor] Share the reasoning splitter and tool constraint across hosts (#42996)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-09 15:19:41 +08:00
37ae292e6f [diffusion] fix: skip LTX-2 SP masks only for globally unpadded sequences (#43134)
Co-authored-by: yatholam <yatholam@amd.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 15:08:41 +08:00
minke.yu 236686454c deploy: bump bench compose templates to 1f6ca2cf6 (#42035+#42194) 2026-10-09 15:06:41 +08:00
Lianmin Zheng c44a42e6d3 docs: reorder README workloads, hardware, and ecosystem (#43307) 2026-10-08 23:46:24 -07:00
e0608d890e fix: use dp_size instead of attn_dp_size for msprobe rank id (#43041)
Co-authored-by: l30044004 <louyujing@huawei.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-09 09:40:43 +03:00
Gabriel Wu 217980566a fix(fa4): isolate sheared-bias workspaces by CUDA stream (#33000) 2026-10-08 23:37:02 -07:00
areneau-cfandJimmy Shong bd2d73daa5 [model] support Clef and Clef-Flash decision models on /v1/systemone (#42721)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-09 14:25:52 +08:00
Michael 51f7ea84c7 [AMD] Keep MiniMax decode-score tile at 128 tokens off gfx950 (#42772) 2026-10-08 23:22:19 -07:00
08fc290f48 [AMD] gfx950 small-batch MoE: expert-count gate for the small sort and FP8 block-scale small-M kernel (#41982)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 23:05:50 -07:00
cba1d07a94 [Memory] Share one full prefill graph output buffer across shapes for tuple and list outputs (#43176)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-08 23:05:29 -07:00
e14ef4b658 [Scheduler] Skip the decode for a request whose last token is already in flight (#42720)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-08 23:00:46 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 1fc6b8962c [Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 (#43083)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:55:35 +08:00
e46aaaa789 [diffusion] feat: accept strided row views in the packed QKV all-to-all (#40588)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 13:55:31 +08:00
Colin Z 3c1703d2ff [AMD] Support Quark DeepSeek-V4-MXFP4 checkpoints on the DSV4 stack, enabling shared-expert fusion and MTP (#31054) 2026-10-08 22:54:49 -07:00
Yuan Luoandluoyuan.luo 9a1aeacc3b [GLM-5.3-Flash] Capture the KDA extend in breakable prefill CUDA graphs (#41105)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-10-09 13:37:20 +08:00
33e7b4cc4b [diffusion] perf: decode plain base64 MiniMax-H3 materials without a per-character loop (#43117)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:25:23 +08:00
Chetan Kumar VermaandClaude Fable 5.1 5d8e98b7fd [XPU][Diffusion] Plain-torch scale-shift for inductor (#41885)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-09 13:07:14 +08:00
saatwiknagpal 6e7beace14 [diffusion] fix: fix int32 offset overflow in the Ulysses packed-QKV pack kernel (#43140) 2026-10-09 11:51:46 +08:00
3ba90f1b52 [rust] Update Dynamo frontend crates to latest compatible releases (#42666)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:46:01 +08:00
9172a73725 [sgl-router] Keep charset=utf-8 on streamed responses (#43232)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:44:52 +08:00
Baizhou Zhang 33bffcd490 [CP] GLM-5.3-Flash Prefill CP implemented on layer boundaries (#42044) 2026-10-08 20:42:11 -07:00
metamergebotandjoshuadeng 7859eff6ca [srt] Inject "__req__" into custom_params only for custom logit processors (#43181)
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-08 20:40:50 -07:00
4ee3f0f370 [Session] Skip the KV free when page alignment empties a streaming-session rewind (#43183)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Joshua Deng <91448271+joshuadeng@users.noreply.github.com>
2026-10-08 20:40:25 -07:00
4e3eaba619 [HiCache] Re-admit a split node's queued buffer-mode write instead of dropping it (#43174)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 20:39:41 -07:00
minke.yu 1f6ca2cf68 test: adopt upstream #42035 auxiliary-output asserts now that checkpoint_disagg_prefill shim exists
build-sglang-image / build (push) Successful in 25m55s
2026-10-09 11:32:33 +08:00
minke.yu bde1fcbf51 test: tolerate chunked kwarg in #42035 polling test override (branch shim keeps chunked flag) 2026-10-09 11:25:05 +08:00
minke.yu c4f3e3be33 refactor: route disagg prefill checkpoints through checkpoint_disagg_prefill shim
Upstream #42035's tests and call structure expect Scheduler.checkpoint_disagg_prefill
(introduced upstream with #41520's checkpoint_kv_cache / insert_req cache API).
This branch still uses maybe_cache_unfinished_req (old cache_unfinished_req API),
so add a thin shim with the upstream name and route the three call sites through it.
No behavior change.
2026-10-09 11:22:42 +08:00
Liangsheng Yin 9ef65b51c2 [mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public (#42194)
build-sglang-image / build (push) Successful in 25m53s
(cherry picked from commit 07e1a4f082)
2026-10-09 11:18:17 +08:00
HEJIAN SANGandClaude Opus 5.5 3e13f1b2c7 [Perf] Serialize OpenAI chat/completion responses in one pass (#42889)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:17:25 +08:00
minke.yu 6ba34507f1 test: adapt #42035 polling test to pre-#41520 name maybe_cache_unfinished_req
build-sglang-image / build (push) Successful in 26m20s
2026-10-09 11:10:39 +08:00
metamergebot 93d610c1fd [PD] Keep ingesting requests while a prefill forward result is pending (#42035)
build-sglang-image / build (push) Successful in 26m6s
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
(cherry picked from commit 6acf2f4736)
2026-10-09 10:56:14 +08:00
ormandjandkpham-sgl 438df9a2a4 [Spec] Fix wrong tokens from compact-verify CUDA graphs when a token tier is captured more than once (#42709)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-08 19:49:23 -07:00
Khoa Pham 8e19db19b0 [KDA] Default safe-gate Blackwell prefill to FlashInfer (#43246) 2026-10-08 19:46:28 -07:00
Chetan Kumar VermaandClaude Opus 5.5 fedab5537b [XPU] Enable ring attention on XPUAttentionBackend (#41406)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 10:41:11 +08:00
MickandMick Qian df1e4f3f2b [diffusion] refactor: share SANA-WM refiner block computation (#42980)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-09 10:38:36 +08:00
Liangsheng Yin f6569bce4d [MemCache] Skip KV events and eviction config on a disabled radix cache (#43253) 2026-10-08 19:27:39 -07:00
Polisetty V R K Jyothendra Varma e54f08c860 [XPU] Misc changes to support XPU (#39817)
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com>
2026-10-09 10:26:46 +08:00
ashwini rathi 51686cf8e7 [XPU] split Intel nightly into kernel-main and kernel-wheel jobs (#41857) 2026-10-09 09:58:39 +08:00
shodoco 86624f14eb [Rust] Pass the media item fps hint to multimodal processors (#43143) 2026-10-08 17:51:06 -07:00
Liangsheng Yin e0ab91e3ca [MemCache] Rename SWAChunkCapPoolConfigurator to SWARequestCapPoolConfigurator (#43202) 2026-10-08 17:50:51 -07:00
b2786a670e [MemCache] Count reclaimable tokens in the unified tree ledgers (#43185)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 17:48:11 -07:00
Liangsheng Yin b4fe282836 [MemCache] Delete ChunkCache (#43201) 2026-10-08 17:31:16 -07:00
Liangsheng Yin 508ba79688 [MemCache] Serve disabled hybrid-SWA models with UnifiedRadixCache (#43200) 2026-10-08 17:30:01 -07:00
Liangsheng Yin 265efe5e01 [MemCache] Serve disabled full-attention models with UnifiedRadixCache (#43199) 2026-10-08 17:27:24 -07:00
Liangsheng Yin 977e2719c7 [MemCache] Serve disabled pure-SWA models with PureSWARadixCache (#43198) 2026-10-08 17:25:54 -07:00
Liangsheng Yin 57ce47862d [MemCache] Report no prefix sharing from a disabled RadixCache (#43197) 2026-10-08 17:23:11 -07:00
Khoa Pham fc3385d530 feat(kda): integrate FlashInfer prefill checkpoints (#41400) 2026-10-08 17:16:33 -07:00
Khoa Pham 1e77db3215 [Perf] Avoid gate copies in gated RMSNorm (#43227) 2026-10-08 17:11:58 -07:00
20d1f3f56a [Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction (#41593)
Signed-off-by: Pei Li <peili@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-08 17:05:04 -07:00
Ankur Singh 2b94cb4eb5 docs(cookbook): add Qwen3.8 GB300 long-context recipe (#43159)
Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
2026-10-08 16:56:30 -07:00
bb43927404 [PD] Charge decode admission the page-rounded allocation when the radix cache is disabled (#43173)
Co-authored-by: hanming-lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 16:48:31 -07:00
Liangsheng Yin 2e64740442 [CI] Avoid Hugging Face Hub API calls when the local model cache is complete (#43229) 2026-10-08 16:27:31 -07:00
Liangsheng Yin baee5f3b62 [CI] Drop the labeled trigger from the extra CI workflows (#43234) 2026-10-08 16:25:50 -07:00
Liangsheng Yin ae4da8cd78 [MemCache] Skip SWA request-cap pool sizing when chunked prefill is off (#43203) 2026-10-08 16:19:35 -07:00
Liangsheng Yin 71584b62c0 [CI] Fix stale fakes in the NVFP4 MoE dispatch and CP strategy unit tests (#43226) 2026-10-08 15:51:39 -07:00
20bf93b6bf [metrics] Call DPBalanceStats.create by keyword in the ratio ladder test (#43218)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 15:24:38 -07:00
2abb761639 [metrics] Count per-rank DP attention pairs and their imbalance (#43180)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 15:01:37 -07:00
metamergebotandHanming Lu 0b658c12ff [Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks (#43188)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
2026-10-08 15:01:16 -07:00
Siyuan Chen b0e06359ac [DSV4.1][PD][5/N] Decode node support DP attention (#40177) 2026-10-08 14:42:16 -07:00
Xuanteng Huang d20bdc1443 Add per-token NVFP4 MoE support for ReLU2 activation (#39199)
Signed-off-by: Xuanteng Huang <xuanteng.huang@outlook.com>
2026-10-08 14:28:22 -07:00
ShunkangzandClaude Opus 5 3d8d44a4a9 [Qwen3.8 CP 3/4] Collocated prefill CP integration (#39723)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-08 14:20:27 -07:00
Sage 5a5ea81036 [rust-server] warm up each rust listener in-process (#43004)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-08 14:09:46 -07:00
e902207a4e [Speculative] Support block verification for the DFlash family (#43012)
Co-authored-by: maocheng <mao.cheng@MacBook-Pro-MD663G60VV.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:59:52 -07:00
d1ba943016 [Model] Add support for EmbeddingGemma 2 (#42782)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:54:45 -07:00
Baizhou Zhang b193ef382a [CI] Clean some unnecessary DSV4 tests (#43195) 2026-10-08 13:53:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad d8448d2e62 [CI] Re-enable GB300 (#43058)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-08 13:23:26 -07:00
metamergebotandhanming-lu 93f2acf614 [Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio (#43187)
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 12:58:58 -07:00
Ming Yang 3fef129ff8 [PD] Allow state-only KV checksum inputs (#42904) 2026-10-08 12:54:03 -07:00
075fd8fa0b [Mamba] Describe the track snapshot dtype by its kernel instead of fp32 (#43192)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-08 12:47:46 -07:00
e3c5719713 [DFlash] Build the attention QKV projection through an overridable hook (#43179)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-08 12:46:28 -07:00
kunchengit 00b3a48753 [dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding (#31773) 2026-10-08 12:44:14 -07:00
Aurick QiaoandQiaolin-Yu b2cb24995d [Debug] Add opt-in NaN detection before sampling (#42690)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:20:57 -07:00
Shenxiu LiuandQiaolin-Yu 393eaed509 [Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather (#34076)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:19:35 -07:00
shodoco 1638c5d699 [http_server] Let custom /generate routes skip the second contract check (#43114) 2026-10-08 11:56:06 -07:00
214347891a [Bugfix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path (#41072)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 18:01:27 +00:00
cbf49695c3 [CI] Add dedicated sglang-processor workflow (#43022)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:57:05 +08:00
shodoco e2fbc6a550 Stream a chunk once stream_interval tokens are unsent (#42643) 2026-10-08 10:12:37 -07:00
Eric Zhang 344b8a863b [Perf] Fuse short-convolution checkpoint tracking metadata (#43013) 2026-10-09 00:26:15 +08:00
Ke Bao 642e81fbb8 Fix DSA index host buffer cleanup (#43100) 2026-10-09 00:21:38 +08:00
Ke Bao 46ad9eb439 Fix K-only host buffer cleanup (#43101) 2026-10-09 00:21:27 +08:00
billishyahao 4ab4d90d78 [AMD][DCP 3/N] add aiter asm for kimi k3 target verify (#34537) 2026-10-08 09:13:03 -07:00
MickandMick Qian 3030e31ff8 [diffusion] refactor: deduplicate request extraction and ComfyUI test helpers (#42903)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 23:44:35 +08:00
黄孝君 9f5d871251 [NPU] Quote the arm64 Triton-ascend wheel URL in npu.Dockerfile (#43035) 2026-10-08 21:44:49 +08:00
943621c336 [sgl-router] Render DeepSeek-V4 through sglang-processor (#42665)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-08 20:16:20 +08:00
11b1fb21de [MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection (#38841)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:39:31 +08:00
6e6ccb26f2 [MiniMax-M3] Instantiate the decode block top-k in register buckets (#38615)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 19:33:17 +08:00
45abea0269 [rust-processor] DeepSeek-V4 parity with SGLang, plus a reusable parity harness and skill (#42664)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:35:30 +08:00
Shangming Cai 9578cb1c63 [CI] Don't run the full GPU suite for renderer-only Cargo.lock changes (#43025) 2026-10-08 18:25:06 +08:00
MickandMick Qian 7dd512b0e8 [diffusion] refactor: deduplicate FLUX positions and RoPE application (#43002)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 18:18:39 +08:00
minke.yu 4d0efcc921 deploy: bring k8s 3p1d RBG cw manifest into git; bump to 1068926cd (merged upstream batch)
Source of truth moves from D:\B300\build-kmctl (untracked) to
deploy/b300-ds41/ds41-3p1d-rbg-cw.yaml. Image carries A-batch (7) + B-batch (4)
upstream cherry-picks + is_hip_runtime import fix; bench-validated on compose
(3p1d 8-conc 160k 98903 tok/s, +50% vs 0928 baseline).
2026-10-08 17:41:06 +08:00
Horace He 8ad24fb560 [Perf] Cache compiled ModelOpt exclusion patterns (#42927) 2026-10-08 17:10:25 +08:00
minke.yu f71fafa684 deploy: bump to 1068926cd (is_hip_runtime import fix) 2026-10-08 16:39:04 +08:00
minke.yu 1068926cd6 fix: import is_hip_runtime in dsv4 low_ratio_compress (cherry-pick #41660 lost it)
build-sglang-image / build (push) Successful in 32m18s
Upstream's file gets is_hip_runtime from sglang.kernels.jit.utils via an
intermediate refactor we don't have; our cherry-pick applied cleanly but
the resulting file references is_hip_runtime() without importing it,
crashing every prefill (NameError in c2_prefill_norm_rope_store).

Also unify Dockerfile.fast with the v3.1 full path (check_deps + --no-deps
delta installs) so fast builds after a kernel-wheel bump don't silently
revert to the base image's older kernels.
2026-10-08 16:31:34 +08:00
minke.yu 48fb49baef deploy: bump cpdspark-nr/b4-dp/3p1d-otlp to dsv41-pd-visioncp-2372cdc06-local-20261008-0808 (merged upstream fixes batch) 2026-10-08 16:21:30 +08:00
Liangsheng Yin 4b384df0c9 [CI] Keep the KL tests' LongBench cache outside the checkout and bound its download (#43068) 2026-10-08 01:17:31 -07:00
Liangsheng Yin 7b9c42cf49 [CI] Remove the unregistered chunked-prefill tests under test/manual (#43081) 2026-10-08 01:16:39 -07:00
Liangsheng Yin 89b5935793 [MemCache] Return only the loaded length from init_load_back; drop empty_device_indices (#43023) 2026-10-08 01:14:33 -07:00
Liangsheng Yin 5a4d01a3ae [MemCache] Return only the matched length from match_prefix; read KV indices off the node path (#42923) 2026-10-08 01:10:14 -07:00
minke.yu 2372cdc06e build: light delta also --no-deps — any dep-graph walk over torch re-downloads pinned nccl==2.29.7 over base's 2.30.7 2026-10-08 16:07:50 +08:00
Spandan Tiwari 2d26823f94 [AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token) (#42152) 2026-10-08 01:06:34 -07:00
minke.yu 7b815cc98b build: COPY check_deps.py from sglang/ tree (context .dockerignore excludes build/) 2026-10-08 16:01:12 +08:00
minke.yu d06c5dde42 build: v3.1 three-class dep delta — kernel wheels (sglang-kernel/flashinfer/sgl-deep-gemm) install via pip --no-deps when pins move
Base ships sglang-kernel 0.4.6.post1 but pyproject pins ==0.4.7; kernel
wheels must track the sglang source, so they are an allowed delta (~370MB
wheel), unlike the torch/nvidia family which must always reuse the base.
Also confirmed 2026-09-24 full build silently downgraded NCCL 2.30.7->2.29.7;
v3 keeps base NCCL untouched.
2026-10-08 15:59:36 +08:00
Cheng Wan b3e6c87be3 [Refactor] Translate each KV loc once per iteration (#42753) 2026-10-08 00:58:28 -07:00
508ddc859a [unified-memory] Admit DFLASH and DSPARK with fused draft KV (#40492)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:56:21 -07:00
c2d491e38b [unified-memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts (#40491)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:55:54 -07:00
minke.yu 8d0ee5e51e build: v3 full path reuses base torch/nvidia via local delta check
build-sglang-image / build (push) Successful in 28m50s
Full-resolution pip walks installed torch's metadata and would re-download
the pinned nvidia family (~6GB, throttled to 2h40m on 2026-09-23) and
downgrade the deliberately newer nvidia-nccl-cu13 2.30.7 to ==2.29.7.
v3 replaces pip full-resolution with check_deps.py: compare pyproject
requirements against base-installed versions locally, pip-install only the
lightweight delta via -r, and fail loudly if a heavy pin (torch*/nvidia-*/
sglang-kernel/flashinfer/cuda-*) becomes unsatisfied. Editable install keeps
--no-deps --no-build-isolation.
2026-10-08 15:54:58 +08:00
Michael 5afe4d6334 [AMD] Fix DSpark draft bucket tests for the raw in-graph metadata default (#42230) 2026-10-08 00:54:26 -07:00
8b80808366 [unified-memory] Fuse the draft model's KV into the target's page envelope (mechanism) (#37628)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:54:19 -07:00
08d30f2300 [unified-memory] Speculative decoding on the unified memory pool: translate every read and write path (#37627)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:52:21 -07:00
33883a5fa3 [AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950 (#41134)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 00:50:16 -07:00
b2cd65b8b0 [AMD] One-launch small-M MoE router for Qwen3.5 on gfx950 (#41133)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 00:47:52 -07:00
metamergebot 75758cb0a0 [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails (#42420)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com>
(cherry picked from commit 4ab720e655)
2026-10-08 15:41:51 +08:00
metamergebot 60e80ac7d1 [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert (#42264)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
(cherry picked from commit 2bae12b9b3)
2026-10-08 15:37:43 +08:00
f795fe352d [Test] Add a docs drift check for server_arguments.mdx (#38520)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:37:22 -07:00
Sasha Sidorov 4ef799aaa6 [PD] Wait for prefill completion before Mooncake early KV transfer (#41395)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c683190b40)
2026-10-08 15:36:21 +08:00
Aurick Qiao 1be762368b [Disagg] Validate state strides before Mooncake transfers (#41607)
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com>
(cherry picked from commit 1a0250c5b6)
2026-10-08 15:35:05 +08:00
DarkSharpness f577b61b52 [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2f6e0ec45d)
2026-10-08 15:32:48 +08:00
DarkSharpness 4792b61332 [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 5916999afb)
2026-10-08 15:32:48 +08:00
DarkSharpness 62f1ed099e [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit d75d5b33f7)
2026-10-08 15:32:47 +08:00
AMD-yanfeiwang bfd9bb2e03 fix(cuda-graph): remove unnecessary CP batch alignment (#40543)
(cherry picked from commit 3dc7f4b816)
2026-10-08 15:32:46 +08:00
metamergebot 6deac260bf [PD] Make the prebuilt last-token H2D copy non-blocking under overlap (#42113)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: bbus <bbus@users.noreply.github.com>
(cherry picked from commit 7dd93b01a5)
2026-10-08 15:32:46 +08:00
Yuwei An 842ad7c4d4 [Fix][DSV4.1] SWA page size with bounded replay (#42128)
(cherry picked from commit ef867fa40d)
2026-10-08 15:32:45 +08:00
Mohammad Miadh Angkad b605d2d370 [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer (#41987)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
(cherry picked from commit 42f2e4fcb7)
2026-10-08 15:32:44 +08:00
Chunan Zeng 785005d21b [DSpark] Mask uncommitted slots before commit-inject cache lookups (#42624) 2026-10-08 00:16:54 -07:00
Yuwei An 54b551df8e [Refactor][TCPCG] Bind live forward batches in BCG eager calls (2/9) (#42286) 2026-10-07 23:55:41 -07:00
MickandMick Qian 5c5f593110 [diffusion] refactor: share KL VAE encode/decode and tiling operations (#42841)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:43:19 +08:00
MickandMick Qian d1b751d958 [diffusion] refactor: inherit existing defaults instead of repeating overrides (#42963)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:42:21 +08:00
74308cf1d6 [diffusion] fix: wait for multimodal process teardown (#40765)
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:36:17 +08:00
4f320fb85a [diffusion] perf: FastH3 8-Step V2 native support, native Blackwell VSA and batched H3 VAE decode (#37662)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-08 14:35:27 +08:00
MickandMick Qian 57e9ff5403 [diffusion] refactor: share timestep retrieval across pipeline stages (#42967)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 14:33:39 +08:00
Liangsheng Yin 0f5bc4af4b [CI] Fix stale fakes in the NPU CUDA-graph backend unit test (#43056) 2026-10-07 23:04:48 -07:00
Mohammad Miadh Angkad ac656ee79a [CI] Re-enable B300 (#42959) 2026-10-07 22:03:16 -07:00
Jzz1943 b7b2975b57 fix(dsa): align FA3 metadata with unpadded queries (#42793) 2026-10-07 21:37:21 -07:00
hmalgewattaandAlex Nails 7327fb225f [DSV4] Guard V4.1 indexer SM100 check on non-CUDA devices (#42114)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-08 12:06:28 +08:00
Thomas Wang b72da1b817 [AMD] Update v4 amd cookbook 1008 (#43033) 2026-10-08 11:18:41 +08:00
AndyLi429andAndyLi429 89bea053ac [NPU] Add NPU cudagraph runner unit tests (#40404)
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com>
2026-10-08 11:18:38 +08:00
chilltongxandchilltongx e7530a3a26 [diffusion] fix: fix forward multimodal token type IDs in image encoding (#34494)
Co-authored-by: chilltongx <284668524+chilltongx@users.noreply.github.com>
2026-10-08 10:55:55 +08:00
WenhaoZhang b0d7254fbc [diffusion] CI: rebase LTX H100 timing references on recent CI runs (#42986) 2026-10-08 10:42:12 +08:00
Jiajun LiandClaude Opus 5.5 99463390f8 [Fix] MiMo-V2: accept checkpoints with the split attention layout (#40978)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 19:18:32 -07:00
c892301ff7 [Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities (#35594)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
2026-10-07 19:16:10 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 7b6b6ee25e [Docs] Cookbook: interactive SemiAnalysis InferenceX AgentX launches for GLM-5.2, DeepSeek-V4 and DeepSeek-V4.1 (#42726)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:33:40 +08:00
MickandMick Qian 21fa2251a0 [diffusion] refactor: share sigma schedules and causal cross-attention (#42848)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:29:33 +08:00
MickandMick Qian 28c08b02c3 [diffusion] refactor: deduplicate MOVA initialization (#42983)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:28:06 +08:00
MickandMick Qian 243f3dc35f [diffusion] refactor: deduplicate Qwen-VL inputs and component accuracy suites (#42860)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:27:22 +08:00
MickandMick Qian 2489bd1d2e [diffusion] refactor: share ModelOpt checkpoint conversion I/O (#42856)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:26:39 +08:00
MickandMick Qian 35608e48d1 [diffusion] refactor: reuse the single-process parallel test fixture (#42964)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:24:53 +08:00
ashwini rathi b91919ffe2 [XPU][diffusion] Re-baseline Wan2.1-T2V-1.3B TextEncodingStage on B60 (#42794) 2026-10-08 09:22:41 +08:00
MickandMick Qian 287b8b793f [diffusion] refactor: deduplicate VAE resampling and LTX tiling (#42866)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:22:11 +08:00
MickandMick Qian 0c00bffc6f [diffusion] refactor: deduplicate Qwen Image RoPE and SenseNova backbones (#42843)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:20:55 +08:00
MickandMick Qian 1e2f524726 [diffusion] refactor: share stacked encoder weight loading (#42988)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:16:03 +08:00
Liangsheng Yin 0427ea8032 [CI] Split the shared Cargo target dir per glibc (#43024) 2026-10-07 18:14:26 -07:00
MickandMick Qian d52ea5ee90 [diffusion] refactor: deduplicate realtime transport and LingBot tests (#42990)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 09:10:10 +08:00
Richard Wang 5cb7fb886c [Feature] support pplx-decider-v1.1 and update the pplx-decider cookbooks (#42645) 2026-10-08 09:03:50 +08:00
MickandClaude Opus 5.5 9101e895ea [diffusion] feat: encode the MP4 while the VAE decodes (MiniMax-H3, Wan) (#41819)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:01:19 +08:00
Cheng Wan 7760d5264a [Refactor] Use destination layouts for model checkpoint mappings (#42590) 2026-10-07 16:24:22 -07:00
Cheng Wan 5e12831cec [Refactor] Load auxiliary and expert checkpoints using owner partitions (#42589) 2026-10-07 16:24:03 -07:00
Cheng Wan 64c7f65eac [Refactor] Retain process groups in parallel linear layers (#42588) 2026-10-07 16:23:40 -07:00
Cheng Wan 267e34516c [Refactor] Select parallel groups for remaining model projections (#42587) 2026-10-07 16:22:57 -07:00
Cheng Wan dec5d9d6fc [Refactor] Let vision encoders select their parallel group (#42586) 2026-10-07 16:22:32 -07:00
Cheng Wan b83827ea02 [Fix] Quantize reloaded FP8 weights after native sharding (#42585) 2026-10-07 16:20:47 -07:00
Cheng Wan 59740a728f [Refactor] Let MLA projections select their parallel group (#42484) 2026-10-07 16:19:33 -07:00
Cheng Wan 961d7290fe [Refactor] Let model MLPs select their parallel group (#42468) 2026-10-07 16:18:14 -07:00
Cheng Wan 4c5130ac5a [Refactor] Let linear layers select their parallel group (#42426) 2026-10-07 16:15:26 -07:00
Liangsheng Yin 1066decce6 [Scheduler] Replace extend_range with extend_end derived from prefix_len (#42822) 2026-10-07 16:07:16 -07:00
Liangsheng Yin 5349d3e9d2 [Scheduler] Derive prefix KV indices from last_node at allocation; drop Req.prefix_indices (#42825) 2026-10-07 16:03:09 -07:00
Liangsheng Yin e9240b3243 [Scheduler] Track prefill progress as prefix_len (#42824) 2026-10-07 15:59:06 -07:00
Kevin MiandClaude Opus 5.5 79ef49cb6b [Docs] GLM-5.3-Flash: measured MI355X MXFP4 agentic recipe (#43015)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:49:49 -07:00
Liangsheng Yin 4bf8f28d15 [Scheduler] Fold match write-back into match_kv_cache (#42823) 2026-10-07 15:43:05 -07:00
Liangsheng Yin b1ffe5c16a [dLLM] Keep the request row across FDFO blocks (#42846) 2026-10-07 15:39:33 -07:00
Ping QiuandZhiqiang Xie 685d4d453d [HiCache] Add SeaweedFS L3 storage backend (#42399)
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-07 15:05:37 -07:00
WenhaoZhang 0b635266d4 [diffusion] CI: warm up CI cases at the request's quality level (#42958) 2026-10-07 23:26:36 +08:00
00b21d6415 [diffusion] fix: reset DiT cache state at the start of each request (#42437)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 23:25:49 +08:00
Mohammad Miadh AngkadandMohammad Angkad 44d10eab5e [GLM-V] Shard the vision tower over the attention-TP group under DP attention (#42931)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-07 23:17:06 +08:00
d9a0f1cb34 [diffusion] fix: fix hybrid SP+TP for I2V models (#41917)
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 22:33:03 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 1e86143bf0 [Bugfix] Return None for empty reasoning_content outside K2 Horizon (#42962)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 21:02:40 +08:00
WenhaoZhang 492dbd0807 [CI] keep diffusion-only kernels and tests out of the srt stages (#42706) 2026-10-07 20:28:53 +08:00
Tianyang GuandClaude Fable 5.1 aa5551d9b6 [diffusion] fix: do not interleave comfy-kitchen MXFP8 scales twice (#38028)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 17:56:39 +08:00
YC Yen-Ching Tseng c389cbaf99 [AMD][diffusion] Route bf16 fuse_scale_shift to the native fallback on gfx1250 (#42859) 2026-10-07 02:37:43 -07:00
YC Yen-Ching Tseng 49d7d60dfa [AMD][diffusion] VAE attention to the math SDPA backend on gfx1250 (#42858) 2026-10-07 02:32:02 -07:00
Yihao WangandClaude Opus 5.5 11972e5207 [deps] bump transformers to 5.19.0 (#42784)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 17:04:08 +08:00
Rain Jiang c52dcc5a01 The sgalng api specs (#41585) 2026-10-07 01:55:01 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 ba77d30256 [FA4] Keep stream last in the SM100 MLA kernel so disk-cached objects load (#42926)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:58:55 +08:00
Khoa Pham e23093077e [DSv4] Fix shared index cache accessor page size (#42832) 2026-10-07 00:53:17 -07:00
jiaryang 1c42ad3679 [ROCm] Skip the tilelang act_quant in the DSA indexer on gfx1250 (#42747) 2026-10-07 00:16:46 -07:00
Shuwen WangandClaude Opus 5 2889158792 [MemCache] test: require every ComponentData field to be classified by each split hook (#42518)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-07 14:53:34 +08:00
paulzhang-tm c721602ad2 Cover SWA admission after declined HiCache restores (#39839) 2026-10-07 14:51:45 +08:00
Hyogeun Oh (오효근)andmickqian 1d019ce1c8 [diffusion] fix: honor the lazy transformer loader contract (#15215)
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
2026-10-07 14:23:00 +08:00
Mark b0dff4009f [diffusion] cli: restore gated Diffusers model detection (#34903) 2026-10-07 14:21:37 +08:00
Mohammad Miadh AngkadandMohammad Angkad f47d89564d [KDA] Default GLM-5-Next spec verify to the CuTe DSL KDA kernel (#42905)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 22:31:41 -07:00
AMD-yanfeiwangandThomas Wang 9ddbba50dd [AMD][DSV4] feat: enable PCP with fp8 unified_kv on gfx950 (#39923)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-06 22:09:18 -07:00
Seas0andMick Qian 43b4abd857 [diffusion] fix: validate SageAttention 2 head sizes without overriding component policy (#41529)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 12:39:54 +08:00
Xinyi Song 168458fd90 [AMD][V4.1] Fix wo_b input flatten for packed fp8 activations on gfx950 (#42796) 2026-10-06 21:35:51 -07:00
Khoa Pham 9e26600f3a [DSA] Clarify pooled-page naming and ownership (#42671) 2026-10-06 21:05:29 -07:00
CuzMi bde01f8dc8 [Diffusion] Fuse Anima AdaLN modulation and cache the FFI callable (#42608) 2026-10-07 12:01:59 +08:00
Tuan Nguyen GiaandCursor c0df52be2d [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path (#34540)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-07 03:20:40 +00:00
Yi (Vincent) ZhongandClaude Opus 5.5 012cd3858b [Fix] Make the fused KDA beta sigmoid bit-identical to torch.sigmoid (#42611)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 20:04:59 -07:00
Thomas Wang 9a21d82958 [AMD] Update v4.1 amd cookbook 1006 (#42847) 2026-10-06 19:58:58 -07:00
Jay Zenith 35f18848db [diffusion] fix: validate output_quality and fix return type annotation (#19530) 2026-10-07 10:53:29 +08:00
Yash AkhauriandBBuf 9c8fc6a8e4 [Bugfix] Preserve empty K2 Horizon reasoning in chat responses (#42623)
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-07 10:44:55 +08:00
d7190b0122 [Perf] Accelerate DiffusionGemma with FA4 and CUDA graphs on Blackwell (#42239)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-07 10:31:00 +08:00
2f6e0ec45d [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-07 09:50:34 +08:00
7a4d6dfd2f [diffusion] feat: name the quality tiers after what they guarantee (#42370)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-07 09:49:30 +08:00
Lianmin Zheng 8f1790f595 Add Rust code owners and grant shodoco CI permissions (#42826) 2026-10-06 18:37:54 -07:00
660b17e8dc [AMD] Let AITER unified verify run without host seq_lens (#42635)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com>
2026-10-06 18:27:27 -07:00
Eddie A. Tsai e13933c4fd [diffusion] docs: add gfx1151 Strix Halo serving guidance (#41670) 2026-10-07 09:14:07 +08:00
Michael 3e914e69d5 [AMD] ci: nightly-test the miles ROCm 10 MI30X image on MI300 (#42771) 2026-10-06 18:05:09 -07:00
WenhaoZhang 6da08fd7cf [diffusion] CI: fix qwen cache-dit step 10/49 perf baselines (#42495) 2026-10-07 08:51:12 +08:00
MickandMick Qian 1351f32bb4 [diffusion] refactor: simplify Kandinsky 6 models, SR sampling and tests (#42768)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-07 08:48:16 +08:00
f1065423d3 [PD] Stop reserving unallocated mamba spec-verify scratch in prefill KV sizing (#42649)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-06 17:16:22 -07:00
Baizhou Zhang f497b626b6 build: use one DeepEP ref for implementation and packaging (#42827) 2026-10-06 17:04:28 -07:00
Duyi-Wang 66dd4e15b9 [AMD][Docs] Add MI355X DeepSeek-V4 Pro PD recipes (#42757) 2026-10-06 16:58:40 -07:00
08aad0c196 [HiCache] Manage buffer-mode backups per storage pool (#42633)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-06 16:45:05 -07:00
jain-riaandishandhanani fd62ee84f7 feat(grpc): expose follower metadata and node-local KV sources (#39659)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-06 16:39:01 -07:00
Chunan ZengandCheng Wan 90aa1cb0fb [Scheduler] Gather prefill-delayer queue timeout across ranks (#42625)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-06 16:15:34 -07:00
Jan BernlöhrandPo-Han Huang 0a0fe001f1 fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends (#41765)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-06 16:02:45 -07:00
6b737fd4c6 Bind the MessageQueue remote socket atomically (#42695)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com>
2026-10-06 15:50:59 -07:00
Trevor Morris 44a2558d40 [NVIDIA] Fix nightly-cu134 build and update (#42661) 2026-10-06 14:52:42 -07:00
264d1c2015 Back the DFlash-family draft KV pool to the post-capture token count (#42651)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-06 14:23:18 -07:00
4506927635 [Metrics] Add per-rank DP attention imbalance metrics (#42755)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-06 14:20:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 38bdf25f8d [Docker] Use uv-managed Python 3.12 and uv for all installs in the CUDA image (#42612)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 14:10:09 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 3eec178e48 [Spec][MegaMoE] Let the speculative draft choose its own W4A4 MXFP4 MegaMoE MMA type (#42022)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:09:57 -07:00
Liangsheng Yin 12004043b8 [Metrics] Read scheduled prefill KV tokens from batch prefix_lens (#42797) 2026-10-06 13:11:29 -07:00
a34cba3c62 [HiCache] Resolve write_back to write_through under buffer_only host memory (#42754)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-06 11:02:23 -07:00
0f57a39152 [rust-processor] Gate render, tokenizer and parser behind cargo features (#42663)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:58:38 -07:00
Byron Hsu abed685de1 [PD] Fix crash for allocation when mamba + pd + decode radix cache (#41927) 2026-10-06 09:36:31 -07:00
MichaelandClaude 209e190ee5 [AMD] ci: build the miles ROCm 10 MI30X image nightly (#41605)
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-07 00:00:11 +08:00
Emil BogomolovandEmil Bogomolov e17935820a [diffusion] fix: fix width/height silently dropped on multipart /v1/videos requests (#35108)
Co-authored-by: Emil Bogomolov <zetyquickly@googlemail.com>
2026-10-06 23:31:31 +08:00
WenhaoZhangandMick 2579875b5d [diffusion] feat: support loading community ComfyUI-GGUF Qwen-Image-2.1 transformers (#42556)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-06 23:30:48 +08:00
49bafb58cd perf(mla): use native SATFINITE FP8 conversion in MLA KV/Q preparation (#42659)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 22:47:47 +08:00
Xiaoshuai Zhangandjet 679d2851ba fix(mimo-vl): keep multimodal features when capturing aux hidden states and correct vision preprocessing (#40888)
Co-authored-by: jet <dev@jetd.one>
2026-10-06 22:15:14 +08:00
16a23a672b [diffusion] model: support Kandinsky 6 (#42743)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Lev Novitskiy <57654885+leffff@users.noreply.github.com>
Co-authored-by: Kirill <kykozlov@gmail.com>
Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
2026-10-06 22:13:29 +08:00
730f1f3e5b [DeepSeek-V4.1] Unify mHC into one state machine and drop medium-batch fusion variants (#42245)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-06 21:01:42 +08:00
WenhaoZhang 2a5f45d3d4 [diffusion] fix: skip zero blocks when hashing large conditioning inputs (#42748) 2026-10-06 20:43:08 +08:00
WMCandjacky.cheng ba94488b42 [diffusion] feat: default Qwen-Image 2.1 to VAE tiling on gfx1151 (#41655)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-06 20:20:58 +08:00
Shangming Cai 1291f3175d [rust-processor] Fix chat-template precedence wording in the processor README (#42756) 2026-10-06 19:10:26 +08:00
Shangming CaiandClaude Fable 5.1 28525b43c4 [sgl-router] Document and reject a zero request timeout (#42751)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 19:07:37 +08:00
cf19ae6d04 [rust-processor] Split sglang-processor into tokenizer, render and parser (#42662)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:59:02 +08:00
Mohammad Miadh AngkadandMohammad Angkad 45db430530 [DSA] Fix pooled page table refresh for target verify CUDA graph replay (#42750)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-06 18:57:52 +08:00
2167cf90c3 [sgl-router] Bound streaming time-to-headers by --request-timeout-secs (retry 5/5) (#42632)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:35:11 +08:00
YC Yen-Ching Tseng f3b5a28f43 [AMD] M3-mxfp8 Nightly Test Update (#42605) 2026-10-06 02:22:53 -07:00
YC Yen-Ching Tseng 542addad32 [AMD] M3-MXFP4 Nightly Test (#42602) 2026-10-06 02:22:01 -07:00
662879e495 [sgl-router] Back off between retry attempts (retry 4/5) (#42631)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-06 17:06:45 +08:00
Gawr Gura 0721 b563bbd597 [Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP (#40671) 2026-10-06 01:58:02 -07:00
e49e971131 [mem_cache] Keep prefix_indices current for chunked requests kept out of the tree (#42467)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-06 01:40:35 -07:00
bb4d0a13c1 [sgl-router] Retry failed dispatches on another worker (retry 3/5) (#42630)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-06 16:28:52 +08:00
Jzz1943 1be3013118 [diffusion] feat: support Ideogram TurboTime LoRA (#30487) 2026-10-06 16:15:21 +08:00
Max Kong 01345197af [diffusion] chore: harden OpenAI upload filename handling (#17270)
=
2026-10-06 16:14:39 +08:00
Jinbao ChenandMick Qian d0494af08b [diffusion] security: pin multimodal-gen ZMQ scheduler ingress to loopback on wildcard --host (#36854)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 16:05:17 +08:00
Liangsheng Yin c7f3ae1dd5 [mem_cache] Hold a request's tree lock as one TreeLock (#42686) 2026-10-06 00:59:38 -07:00
05340e9ecb [sgl-router] Make one request dispatchable more than once (retry 2/5) (#42629)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:24:02 +08:00
c683190b40 [PD] Wait for prefill completion before Mooncake early KV transfer (#41395)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-06 15:17:12 +08:00
8c273d6813 [sgl-router] Let selection skip workers a request already failed on (retry 1/5) (#42628)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:10:51 +08:00
Rumit Desai 33ce0943e9 [Bugfix] Guard merge_state sgl_kernel import on non-CUDA platforms (#42682) 2026-10-06 14:19:20 +08:00
Joe 40fb5935c0 Allow platforms to declare Mamba extra-buffer support (#36410) 2026-10-06 14:18:49 +08:00
Liangsheng Yin 9a09b48da7 [Fix] Add maybe_hand_to_session to the one-batch benchmark TreeCacheNamespace (#42723) 2026-10-05 23:09:41 -07:00
Mohammad Miadh AngkadandMohammad Angkad 60878c3fba [CI] Temporarily disable B300 (#42704)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-05 22:27:54 -07:00
b524de2de6 [diffusion] optimization: release the allocator cache when the server idles not per request (#41718)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 12:27:08 +08:00
e446698cdd [SM120&90] Add CUDA fused Triton sparse-MLA prefill backend for DSA (#32779)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-05 21:08:05 -07:00
1e1bdeedbb [Perf] Improve GLM-5.3-Flash DFlash decoding on B300 (#42489)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 12:05:37 +08:00
Mohammad Miadh AngkadandMohammad Angkad fcbbefa82a [GLM-5.3-Flash] Autotune MoE kernels at prefill sizes (#42693)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-05 21:04:54 -07:00
Yuwei An 620c76f25d [DSV4.1] Add opt-in TRT-LLM sparse attention support (#41603) 2026-10-06 12:02:10 +08:00
Zhang, Jiejing c76faa1ff2 docs(cookbook): enable Top-K v2 for GLM-5.2 MI355X MTP (#42613) 2026-10-06 11:33:24 +08:00
Ke Bao ed75fe67f1 Declare hybrid Mamba indexer host pool (#40915) 2026-10-06 10:18:36 +08:00
CuzMi 67959f80ab [diffusion] optimization: fuse Anima FP32 split-half Q/K RoPE (#42609) 2026-10-06 10:18:09 +08:00
Thomas Wang 1abd356595 [AMD] Add engram host/device address mapping (#42552) 2026-10-05 19:12:12 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 f2f0be731c [Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings (#42167)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:40:30 +08:00
MickandMick Qian 21c9bbdf2c [diffusion] fix: fix concurrent server perf dump writes across ranks (#41541)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-06 09:29:51 +08:00
Yash AkhauriandXiaoyu Zhang 36b216e2ee [Docs] Refresh K2 Horizon checkpoint revisions (#42130)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-06 09:19:04 +08:00
5d385007b3 [diffusion] kernels: take the Wan VAE decoder's remaining first-decode work off the request path (#41830)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:05:29 +08:00
3120bb87a2 [diffusion] fix: fix the OOM hint's placement advice and add a 24 GB Wan2.1 recipe (#41723)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:01:02 +08:00
Liangsheng Yin ad7a9b53cc [mem_cache] Make release_kv_cache's final checkpoint an explicit checkpoint flag (#42270) 2026-10-05 17:32:07 -07:00
Po-Han Huang (NVIDIA) 62af5f56a5 [CI] Drop the redundant and broken tc_piecewise tests (#42571) 2026-10-05 17:29:32 -07:00
Liangsheng Yin c01d59eb90 [Session] Let a streaming session own its KV record and tree lock from the first turn (#42537) 2026-10-05 17:28:09 -07:00
Liangsheng Yin f3fe753475 [misc] Rename SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA (#42657) 2026-10-05 17:24:15 -07:00
Sage 4409d2fd22 [ci] add renderer automatic labeler (#40923)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-05 17:12:51 -07:00
Yuwei An 18ec2ecce3 [Refactor][TCPCG] Relocate shared graph tensor and DSA head-gate helpers (1/9) (#42285) 2026-10-05 17:06:52 -07:00
ADiL 968726f3ce [HiCache] Route --file-storage-path to the file storage backend (#33883)
Signed-off-by: adlashab <adlashab@amd.com>
2026-10-05 15:33:49 -07:00
61ba98443d [DSA] k-pool: 256-token logical page so index page id = logical page id (#42178)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-10-05 15:02:44 -07:00
c765f8818a [DFlash2] Keep the greedy selector walk in range for all-NaN score rows (#41737)
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 14:29:13 -07:00
maocheng23 57826a62ea [Speculative] Add opt-in block verification (#42297) 2026-10-05 11:49:23 -07:00
70f0b7351e [sgl-router] Improved policy balanced mode (#42545)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-05 09:48:37 -07:00
debanshd efb62ce269 [diffusion] fix: resolve QKV layout inference for MiniMax-H3 checkpoints (#40158)
Signed-off-by: Debanshu Das <debanshu@google.com>
2026-10-05 23:01:37 +08:00
150f568900 [diffusion] kernels: stop specializing on per-request sequence lengths (#41710)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 22:21:01 +08:00
a977e3b9d5 [diffusion] optimization: stream an oversized DiT in auto mode instead of OOMing (#41721)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:58:27 +08:00
ad3e80d4de [diffusion] chore: warm the image writer and the HTTP route models at startup (#41833)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:58:00 +08:00
c83c84b422 [diffusion] chore: keep the DiT off component offload under explicit multi-GPU fsdp (#41722)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 21:56:52 +08:00
wxy 23bf107469 [diffusion] optimization: cache LTX-2 RoPE coords to avoid per-step recompute (#22441) 2026-10-05 21:21:36 +08:00
Shiwen Cheng 91f9bf8d17 [diffusion] fix: fix scheduler host not working when ipv6 in multi modal gen (#22813) 2026-10-05 21:14:09 +08:00
liukeandMick f20f8e120d [diffusion] model: restore folded H3 GGUF patch embedding (#39523)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-05 21:10:06 +08:00
MickandMick Qian 83a6e1d39b [diffusion] docs: keep model compatibility details in cookbook recipes (#42227)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 20:02:43 +08:00
Akshat Anandandcipheraxat 860ebe720e [diffusion] fix: skip warmup preferred preload when it would OOM (#40761)
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
2026-10-05 19:59:35 +08:00
aiyueqiandMick 8663e3b670 [diffusion] optimization: avoid full-video clone in VAE post-processing for MiniMax-H3 (#40095)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-05 19:56:11 +08:00
d349e672bc [diffusion] feat: recycle warmup-only entries of conditioning-cache at the first served store (#41832)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:55:30 +08:00
7352073ec6 [diffusion] feat: support --async-output-save to overlap output saving with the next request (#37549)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 19:53:38 +08:00
WenhaoZhangandMick Qian f048d5aa4b [diffusion] feat: add community LoRA recipes and Kohya mapping (#35857)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 18:22:56 +08:00
08fa37e48d [sgl-router] Add reorg bucket config with per-group admission (#42429)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-05 18:17:32 +08:00
Shangming CaiandClaude Opus 5.5 ab4f8a44f4 [sgl-router] Point the design doc at the shared engine ranking (#42572)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:16:42 +08:00
a96b146491 [sgl-router] Move pressure and prefix signals out of legacy policies (#42428)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:10:29 +08:00
mrusanovskyandPo-Han Huang 26a1377c50 [Spec] LiLiCorr: named head MLPs and quantized head linears (#42057)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-05 02:43:59 -07:00
WenhaoZhangandMick Qian 284cda01fb [diffusion] optimization: fuse the video VAE decoder's RMSNorm and QK RoPE for MiniMax-H3 (#41906)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 17:30:39 +08:00
0263fdacb2 [diffusion] fix: keep generic warmup valid for step-floored models and stop reporting a failed warmup as warm (#41641)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:23:06 +08:00
elvischenvandClaude Opus 5.5 55dcf3254f Fix XQA draft extend with FP8 KV cache in trtllm_mha (#41656)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 01:37:54 -07:00
Xiaoyu Zhang abd6c374c7 [Perf] Optimize MiMo local audio attention on Hopper (#42101) 2026-10-05 16:14:56 +08:00
Michael b3f967e809 Revert "Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)"" (#42558) 2026-10-05 16:12:21 +08:00
Aleksi Vesantoandjacky.cheng 9f7d38ffd3 [diffusion] attention: Allow disabling sequence masking in SP (#33855)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-05 15:22:53 +08:00
Aleksi Vesantoandjacky.cheng c41b2d26e5 [diffusion][ROCm][Perf]: Set gfx942 AITER FMHA rounding mode to rtz instead of rtna (#28650)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-05 15:21:04 +08:00
MickandMick Qian dd2ffdb680 [diffusion] nightly: keep nightly regression baselines on a comparable methodology (#42226)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 15:19:21 +08:00
438d9d2074 [Feature] DeepEP v2: expanded (do_expand=True) prefill dispatch (#37261)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-10-05 00:02:10 -07:00
cafb37a10d [JIT] Fuse FP8 KV-cache quantization into the prefix-valid commit kernel (#40785)
Co-authored-by: Jayden Yu <jaydenyu@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-05 14:47:54 +08:00
WenhaoZhangandMick Qian f218798006 [diffusion] CI: relax the load latency check and extend the 2-GPU retry deadline (#42505)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-05 14:43:27 +08:00
520d6d0229 [DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs (#41659)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-05 14:33:37 +08:00
Liangsheng Yin 503b898540 [Session] Count streaming session KV by owner in pool accounting (#42539) 2026-10-04 22:15:57 -07:00
Michael 456d533e61 Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)" (#42541) 2026-10-04 21:59:12 -07:00
Liangsheng Yin 1818db4e54 [CI] Drop the echoed command from /rerun-test result replies (#42543) 2026-10-04 21:50:53 -07:00
ashwini rathi b1bbd74f28 [XPU][CI] Disable test_xpu_graph until tc_piecewise is removed (#41634) (#42524) 2026-10-05 11:09:26 +08:00
Xiaoyu Zhang f385390be5 [Refactor] Simplify GDN, vision RoPE and MLA kernel dispatch (#42506) 2026-10-05 09:37:53 +08:00
734cf3cf3b [PD] Reload NIXL peer metadata after the peer is invalidated (#41691)
Co-authored-by: agent <agent@local>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-05 00:49:26 +00:00
Yuwei An f70e8c68fc [Dsv4.1] Bounded replay with sparse mla path (#42273) 2026-10-04 17:19:14 -07:00
Eric.Chin.AMDandBingxu Chen b792228b35 [ROCm] topk v2: split one long row across blocks, the CDNA cluster-path equivalent (#39931)
Co-authored-by: Bingxu Chen <bingxche@amd.com>
2026-10-04 16:54:37 -07:00
Cheng Wan 1f299e9f2f [Refactor] Build layer stacks in order with append_stages (#42481) 2026-10-04 16:40:24 -07:00
Cheng Wan a99567ab74 [Fix] GigaChat 3.5: build the stage boundaries once (#42480) 2026-10-04 16:39:34 -07:00
Cheng Wan 5c8fb9e938 [Fix] Offer the fused MoE finalize all-reduce only to a block that owes the sum (#42479) 2026-10-04 16:39:06 -07:00
Cheng Wan f07c1a398e [Refactor] Build the Qwen4 experimental decoders from stage boundaries (#42478) 2026-10-04 16:38:36 -07:00
Cheng Wan 92d60351e2 [Refactor] Build the Hunyuan V4 decoder from stage boundaries (#42477) 2026-10-04 16:38:03 -07:00
Lianmin Zhengandjiayisuse f5ccc4d7d9 [Kernel] Use explicit reciprocal scaling for recurrent Q/K normalization (#42486)
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-04 15:30:03 -07:00
Tarang Khanna d475b5a25c [weight_cache] Coordinate daemon readiness across ranks (#38828) 2026-10-04 15:20:13 -07:00
Aditya Sharma caf96324fc [Apple Silicon] Add an Apple Silicon (MPS) platform to the standard Torch model runner (#36780) 2026-10-04 15:03:40 -07:00
SuperSongandCheng Wan bab04cd791 fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots (#42072)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 12:51:22 -07:00
Mohammad Miadh AngkadandMohammad Angkad fea3c088a1 [CI] Temporarily disable GB300 (#42463)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 12:24:04 -07:00
Zhang, Jiejingandkyle-256 aa0ce20201 [ROCm] GLM-5.2 decode path: decode-shaped MoE/MLA tiles, split speculative softmax, and bf16 GEMM routing (#41725)
Co-authored-by: kyle-256 <Kyle.Zhao@amd.com>
2026-10-04 11:19:37 -07:00
Xiaoyu Zhang 7a719e9a65 [AMD] Fix RoPE cache dtype and diffusion CI failures (#42494) 2026-10-05 00:36:26 +08:00
billishyahao 0343bc1483 [AMD][Docs] fix mori io and umbp playground (#42512) 2026-10-04 09:28:11 -07:00
Shuwen Wang 137c084000 [HiCache] refactor: retire in-flight storage prefetches through one helper (#41453) 2026-10-05 00:13:07 +08:00
WenhaoZhang affa261e3d [diffusion] feat: enable cuda graphs for observation encoding and denoising steps for action models (#42171) 2026-10-04 22:22:08 +08:00
WenhaoZhang 279d3f3888 [diffusion] feat: load ComfyUI/ai-toolkit fused gate_up LoRAs and diffusers metadata alpha for qwen-image-2.1 (#42174) 2026-10-04 22:21:13 +08:00
WenhaoZhang 44fac689f5 [diffusion] optimization: decode json pixel lists to numpy at the entrypoint (#41895) 2026-10-04 22:20:20 +08:00
Michael ab9d8f029c [AMD][DI][CI] mi355x spur: skip RDMA ports that are not PORT_ACTIVE; dump driver log on failure (#41405) 2026-10-04 21:23:54 +08:00
Shuwen Wang 35f3c96ff4 [CI] Fix bootstrap sender registration test on main (#42487) 2026-10-04 18:45:31 +08:00
Shangming CaiandClaude Opus 5 148da8e77d [PD] Validate decode state layout once at registration (#42051)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 18:32:30 +08:00
Xiaoyu Zhang bc5a055dd7 [diffusion] move kernel validation into launchers and simplify dispatch (#42391) 2026-10-04 18:23:29 +08:00
Michael 29bcb74a89 [AMD][CI] Fix ROCm kernel wheel sources: drop eagle_utils, add DSV4 kernels (#42461) 2026-10-04 17:54:28 +08:00
SuperSongandCheng Wan 6cec8f98ce [Bugfix] Fix unified-memory compaction gates and pending page reuse (#39982)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 02:48:08 -07:00
SuperSongandCheng Wan 1d02a36bb7 fix(inkling): translate unified-memory checkpoint destinations to physical slots (#38229)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-04 02:45:45 -07:00
WenhaoZhangandClaude Opus 5.5 1e490772e5 [diffusion] fix: pin the JoyAI-Echo overlay's source revision (#42471)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 17:19:09 +08:00
ba68306256 [diffusion] feat: let requests choose the video libx264 preset (#41797)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 16:22:36 +08:00
Tokha233andWenhaoZhang fbce0d9478 [diffusion] fix: load serialized h3 INT8 in ComfyUI integrated mode (#42121)
Signed-off-by: Tokha233 <61346912+Tokha233@users.noreply.github.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
2026-10-04 16:08:56 +08:00
Mohammad Miadh AngkadandMohammad Angkad 0bdd0cc8dc [Deps] Upgrade the CUDA PyTorch stack to 2.14 (#38641)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 00:48:02 -07:00
Kotthagattu Meher Sai 9631f6f2d0 [XPU] Enable fused QK-norm + RoPE for Qwen3-MoE (#40190) 2026-10-04 15:45:18 +08:00
Mohammad Miadh AngkadandMohammad Angkad 7eb628612c [Fix] Publish the parallel config the CLIP attention test reads (#42460)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-04 00:44:51 -07:00
Baizhou Zhang 55c5415c17 [CP] Support scattered interleave CP inputs with expert parallelism (#42041) 2026-10-04 00:43:31 -07:00
937c0a6cfe Fix unified HiCache physical transfers (#39479)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-04 00:38:56 -07:00
billishyahao 6cc661f223 [AMD][DCP 2/N] Enable aiter asm ps for kimi k3 prefill (#34438) 2026-10-03 23:34:45 -07:00
Liangsheng Yin 048b2a07f3 [Session] Release aborted streaming turns through the normal release path (#42330) 2026-10-03 22:30:25 -07:00
Mohammad Miadh AngkadandMohammad Angkad 1315d171a0 [Diffusion][CI] Add missing B200 Cirrascale runner E2E baselines (#42329)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:10:32 -07:00
Brayden ZhongandMohammad Angkad a3275295c4 [MoE] Use FlashInfer A2A for prefill instead of AG+RS (#39818)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:09:00 -07:00
Mohammad Miadh AngkadandMohammad Angkad c12bc635ca [CI] Install only userspace libgdrapi for GDRCopy (#42233)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:05:35 -07:00
Mohammad Miadh AngkadandMohammad Angkad 0e6d7eba5f [Fix] Count attention-DP ranks in one-batch bench running cap (#42331)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 21:04:27 -07:00
Baizhou Zhang fccc61c1c5 [CP] Support DP x TP x CP for attention with interleave CP strategy (#42038) 2026-10-03 20:57:36 -07:00
Aurick Qiao f24db66729 [PD] Publish prefill DP rank whenever forced lookup is enabled (#41786) 2026-10-04 11:46:20 +08:00
metamergebotandmetamergebot 169e6223ef [PD] Drop the removed MooncakeKVSender args from test_prefill_complete_integration (#42454)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-03 20:29:06 -07:00
4611ea365e [HiCache]: Fix host lock ownership across radix splits (#38480)
Signed-off-by: Lukas Humbel <lhumbel@nvidia.com>
Co-authored-by: Jialin Ouyang <jialino@meta.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-04 11:16:04 +08:00
ff219e159e [AMD] MiniMax-M3: fuse sparse QK norm, RoPE and cache writes with AITER (#35357)
Co-authored-by: zijiec <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-03 19:49:53 -07:00
sglang-botandsglang-bot 8afef58af4 docs: sync LMSYS SGLang blog cards (#41296)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-04 02:26:05 +00:00
5606fb8592 [Fix] HiCache: carry registered Mamba slot side states (Qwen4-Exp PLE) through the host tier (#39862)
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
2026-10-04 10:15:59 +08:00
Michael f2385314b2 [AMD] Add DeepSeek-V4.1-Flash MI35x nightly accuracy test (#41476) 2026-10-03 18:16:44 -07:00
Xinyi Songandkk 663984b5a1 [AMD] Keep aiter's tuned GEMM for the DeepSeek-V4 compressors (#41931)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
2026-10-03 18:11:37 -07:00
trebladev 295a53c90e [diffusion] optimization: fuse Flux3 rowwise FP8 quantization with triton (#41671) 2026-10-04 09:01:17 +08:00
71dc292b97 [AMD] Use AITER ASM prefill for MiniMax-M3 HD128 attention (#41707)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-03 17:54:49 -07:00
4ab720e655 [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails (#42420)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com>
2026-10-03 16:01:32 -07:00
metamergebotandmerrymercy 4301be7bdb Pick the default nccl_port below the kernel ephemeral range (#42405)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
2026-10-03 16:00:53 -07:00
Cheng Wan 10edb05e72 [Refactor] Let the remaining placement consumers read the parallel context (#42348) 2026-10-03 14:59:32 -07:00
07064fd4a4 [sgl-router] Credit routed prompts before their KV events arrive (#42276)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-03 14:57:49 -07:00
Cheng Wan b2aa993cb8 [Fix] Skip a draft's decode recapture when it owns no decode graph (#42347) 2026-10-03 14:54:12 -07:00
Cheng Wan 1a18de4b29 [Fix] Load a draft's tensor weight update from its deployment TP rank (#42346) 2026-10-03 14:52:25 -07:00
sglang-botandsglang-bot 061e712bab chore: bump sgl-kernel version to 0.4.9 (#42427)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-03 14:29:04 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3122b9c995 Update AOT kernels for Torch 2.14 (#42366)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-03 14:26:34 -07:00
Liangsheng Yin df53c978f1 [mem_cache] Run mamba models on UnifiedRadixCache when the radix cache is disabled (#42354) 2026-10-03 14:11:13 -07:00
Liangsheng Yin 3e9a120add [Fix] Make Hf3fsMockClient reads and writes thread-safe with pread/pwrite (#42425) 2026-10-03 14:02:24 -07:00
8bf780dda5 feat(metrics): expose deferred decode KV release metrics (#41128)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-04 04:52:21 +08:00
Yanbin Jiang 749a89abd6 [LoRA] Reorganize kernels and add CODEOWNERS (#42299) 2026-10-03 13:43:40 -07:00
fd5e68f99e [sgl-router] Repair KV-event sequence gaps from the engine's replay socket (#42275)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
2026-10-04 04:43:17 +08:00
aaf7ebe077 Advertise the KV-event replay endpoint in /server_info (#42274)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 04:42:33 +08:00
AMD-yanfeiwang 3dc7f4b816 fix(cuda-graph): remove unnecessary CP batch alignment (#40543) 2026-10-03 12:14:28 -07:00
AMD-yanfeiwang e83c95b650 [AMD] Fix AITER weight slicing for CP decode TP (#39950) 2026-10-03 12:09:08 -07:00
AMD-yanfeiwang f327f92424 [AMD] Port CP V2 to the DeepSeek-V4 HIP backend (#34200) 2026-10-03 12:08:17 -07:00
6acf2f4736 [PD] Keep ingesting requests while a prefill forward result is pending (#42035)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-03 12:02:21 -07:00
b016ca406f Support post-capture KV sizing for the unified hybrid-SWA pool (#41961)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-03 09:54:20 -07:00
2bae12b9b3 [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert (#42264)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-03 09:45:56 -07:00
5b5d721239 [PD] Add opt-in prefill-complete decode KV allocation (#40703)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-03 21:03:09 +08:00
5916999afb [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-03 19:31:05 +08:00
d75d5b33f7 [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
2026-10-03 18:25:22 +08:00
DarkSharpnessandClaude Opus 5 379ec90fb2 [DSv4.1] Fold q_rope_store into fused_q_norm_rope (#41657)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-03 18:24:41 +08:00
Cheng Wan 7be5e3473c [Refactor] Drop the reduction-skip mechanisms stage boundaries no longer use (#42312) 2026-10-03 02:21:55 -07:00
Cheng Wan 71b04e02cf [Refactor] Build the ZAYA1, IQuest-Q1 and Gemma 4 decoders from stage boundaries (#42311) 2026-10-03 02:21:34 -07:00
Cheng Wan f85c2c4923 [Fix] GigaChat 3.5: apply the sandwich norms to the complete sums (#42310) 2026-10-03 02:21:10 -07:00
Cheng Wan 6faccf3e84 [Refactor] Build the GLM-4, GLM-Image and Granite MoE hybrid decoders from stage boundaries (#42309) 2026-10-03 02:20:40 -07:00
Cheng Wan e4554fd5e5 [Refactor] Build the Llama and Nemotron-NAS decoders from stage boundaries (#42308) 2026-10-03 02:20:18 -07:00
Cheng Wan 740a4d5955 [Refactor] Build the Qwen2 decoders from stage boundaries (#42307) 2026-10-03 02:19:56 -07:00
Cheng Wan 2c739b225b [Fix] Jet-Nemotron build and Granite MoE hybrid final norm (#42306) 2026-10-03 02:19:33 -07:00
Cheng Wan b7fc516986 [Fix] EXAONE MoE under DP attention and DeepEP (#42305) 2026-10-03 02:19:19 -07:00
Cheng Wan 2fa17a6467 [Refactor] Build the ERNIE 4.5 VL MoE and EXAONE MoE decoders from stage boundaries (#42304) 2026-10-03 02:18:56 -07:00
Cheng Wan f89ead192f [Fix] EXAONE and ERNIE 4.5 VL MoE architecture, backend and PP issues (#42303) 2026-10-03 02:18:33 -07:00
Cheng Wan be20e7453c [Fix] Step-3.5 DeepEP routed scaling and Sarvam shared expert under dense TP1 (#42302) 2026-10-03 02:18:11 -07:00
Cheng Wan b157d548bb [Refactor] Let stage boundaries complete every stage-output sum (#42301) 2026-10-03 02:17:48 -07:00
Cheng Wan d5a8d76bf5 [Refactor] Drop TBO op methods that no strategy ever schedules (#42300) 2026-10-03 02:17:26 -07:00
Liangsheng Yin 122b59152c [mem_cache] Replace is_chunk_cache / is_tree_cache with supports_prefix_sharing (#42362) 2026-10-03 02:07:55 -07:00
Banruo Liu 8293f9af54 [PD] Centralize drain-aware abort acknowledgements (#37077) 2026-10-03 16:51:11 +08:00
saatwiknagpal 1093c501df [diffusion] optimization: route subBlock sparse attention in head chunks (#41985) 2026-10-03 15:54:54 +08:00
Yanbin Jiang 254aa41f78 [Fix] Guard DeepSeek NVFP4 shared-expert fusion for LoRA and FP4 backends (#42203) 2026-10-03 00:37:57 -07:00
Tokha233 ce617d410b [diffusion] fix: fix native FP8 format handling for FLUX 3 row-wise linears (#42255) 2026-10-03 15:23:41 +08:00
Mohammad Miadh AngkadandMohammad Angkad 00bcc25f6c [HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag (#42215)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 22:27:57 -07:00
Jimmy Shong 0072e2cc08 [Fix] Validate per-layer rope_parameters on transformers 5.17 (Laguna-S/XS-2.1 nightly) (#42214) 2026-10-02 21:49:45 -07:00
Jacob0226andThomas Wang af1bef3eaf [AMD] GLM-5.3-Flash: fuse shared expert and KDA projections on Quark MXFP4 (#41870)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-02 20:47:47 -07:00
Richard WangandRichard Wang 482b062d28 [Feature] serve pplx-decider decision checkpoints on /v1/systemone (#42183)
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com>
2026-10-03 10:18:53 +08:00
WenhaoZhangandXiaoyu Zhang 41c6ffea89 [diffusion] fix: keep residual_gate_add on the JIT CUDA path for cont… (#42240)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-03 10:12:11 +08:00
Xiaoyu Zhang 391e665dab [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (#41251) 2026-10-03 10:04:48 +08:00
MickandMick Qian 65c7425a56 [diffusion] fix: respect explicit residency over pipeline preload hints (#41622)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-03 09:54:21 +08:00
Liangsheng Yin a8b54f138e [Session] Run streaming sessions only on UnifiedRadixCache; reject unverified tree caches (#42295) 2026-10-02 18:15:59 -07:00
8ff02f5820 [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) (#41729)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
2026-10-02 17:57:42 -07:00
Kevin MiandClaude Opus 5 65f759144d [HiCache] Let the MiniMax-M3 K-only host pool join a host pool group (#42166)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-02 17:00:13 -07:00
Baizhou Zhang 3bde8eb2d8 [CP 1/5] Remove the CP adapter and separate interleave transport from boundary reduction (#42036) 2026-10-02 16:50:20 -07:00
Zhang, Jiejing cbe070b6bf [Docs][AMD] Update GLM-5.2 MI355X daily image to 20260930 (#42278) 2026-10-02 15:39:07 -07:00
b69a5f296a [AMD] Add opt-in MiniMax-M3 TP4 indexer context partitioning (#41488)
Co-authored-by: ThomasNing <thomas.ning@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-10-02 15:34:29 -07:00
fa7784d140 [unified-memory] Copy page envelopes in place during compaction (#42252)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-02 15:12:14 -07:00
e8fab85a02 [KDA] Add an opt-in decode-parity mode to the Triton multi-token recurrence (#42251)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-02 14:59:56 -07:00
19794f5743 [sgl-router] Add SGLang's /v1/classify (#42258)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:56:16 +08:00
Liangsheng Yin 8a328e867b [mem_cache] Rename the tree's request-level insert_req to checkpoint (#42202) 2026-10-02 14:42:35 -07:00
2ec9e3b11d [eplb] Warm default-group NCCL P2P transports before KV-cache sizing (#42026)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: luccafong <luccafong@users.noreply.github.com>
2026-10-02 14:34:55 -07:00
ebf161cea4 [sgl-router] Add SGLang's /v1/rerank (#42205)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 05:33:40 +08:00
Kan WuandKan Wu 8c9030ce3f [sgl-router] Unify reorg session and cache affinity modes (#41990)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-03 05:21:22 +08:00
Yuhao Wang 0098b9d29d [Spec] Fix Qwen3.5 text model EAGLE3/DFLASH aux-layer capture (#42002) 2026-10-02 14:05:01 -07:00
raghotham a64725d407 [HiCache] Fall back to host memory when no cgroup fs is mounted (#42120) 2026-10-02 13:59:40 -07:00
7dd93b01a5 [PD] Make the prebuilt last-token H2D copy non-blocking under overlap (#42113)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: bbus <bbus@users.noreply.github.com>
2026-10-02 13:48:39 -07:00
Liangsheng Yin 07e1a4f082 [mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public (#42194) 2026-10-02 13:46:32 -07:00
354ed2a7b6 Size the VMM graph-input exchange by the widest input across ranks (#42125)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: oulgen <oulgen@users.noreply.github.com>
2026-10-02 13:45:32 -07:00
4a4dad1cf1 [Metrics] Label scheduler stage wall time by sampled forward overlap (#42025)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-02 13:43:53 -07:00
ChangLiu0709andNikolai Protasov f6fcda8827 [AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel (#41282)
Co-authored-by: Nikolai Protasov <289648307+amd-nprotaso@users.noreply.github.com>
2026-10-02 11:57:03 -07:00
Shuwen Wang 89f21671bb fix: preserve QSA indexer state through HiCache (#39893) 2026-10-02 23:49:38 +08:00
kkandwunhuang a74f259f38 [AMD][V4.1][*/N] Fuse MXFP8 activation quant into producer kernels on gfx950 (#42055)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-02 08:34:44 -07:00
Alex O. P.andPo-Han Huang e667ab10d5 [diffusion] quantization: ConvRot INT8 online W8A8 for DiTs (sgl-kernel + convrot_int8_customkernel) (#38040)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-02 23:26:28 +08:00
Xiaoyu Zhang 1a2eb6c426 [Kernel] Use Cake softmax for qualified SM103 sampling workloads (#42004) 2026-10-02 23:06:43 +08:00
bb9a820f09 [diffusion] chore: make perf dump one writer per replica with no first-request NCCL setup (#41711)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:44:10 +08:00
Michael 2cd92d10b9 [AMD] Skip the AITER #6042/#5967 patches on gfx1250 (#42219) 2026-10-02 21:10:01 +08:00
edee4308bc [sgl-router] Add OpenAI /v1/embeddings (#42191)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:18:24 +08:00
1330e437e1 [sgl-router] Add native /generate endpoint (#42147)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:09:04 +08:00
Michael e020b389b6 [AMD][CI] Partition stage-b-test-1-gpu-small-amd-mi35x to stop the 30-min timeout (#41973) 2026-10-02 19:29:20 +08:00
MickandMick Qian 0f71ec8656 [diffusion] UX: deduplicate SM120 FP8 fallback warnings across layers (#41542)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-02 19:25:15 +08:00
Michael 544e46c080 [AMD][CI] Fix VLM MMMU nightly max_tokens for CoT prompt (#41979) 2026-10-02 18:59:05 +08:00
YAMYandPo-Han Huang 895525f062 [Mamba] Warm cache COW kernel before serving (#36266)
Signed-off-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-02 18:52:37 +08:00
Alex 1db33bc367 [PD][NIXL] Fix prepared dlists for KV entries of different lengths (DeepSeek-V4) (#42005) 2026-10-02 17:59:31 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 41bd213c5f [Test] Add get_swa_key_page_size to the Q8KV8 sparse-prefill fake KV pool (#42211)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 16:00:15 +08:00
Mohammad Miadh AngkadandMohammad Angkad d954c88a4f [Fix] Update stale source-patch match in dumper comparator e2e test (#42179)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 00:37:37 -07:00
Mohammad Miadh AngkadandMohammad Angkad e7d9c0299b [Docs] Keep the last CUDA 12 image tag fixed during install version bumps (#42201)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 00:17:13 -07:00
sglang-botandsglang-bot d61a9c28ba chore: bump docs install version to 0.5.21 (#42013)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-10-01 23:16:26 -07:00
Liangsheng Yin 50be533d09 [mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes (#41520) 2026-10-01 21:41:45 -07:00
MingyiandMick Qian b906cd3f45 doc: refresh README (#41967)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-02 12:11:33 +08:00
saatwiknagpal bf8adf9602 [diffusion] fix: fix int32 offset overflow in SubBlock router kernels (#41965) 2026-10-02 11:22:53 +08:00
Khoa Pham 6be84e1b72 [Fix] Select the Marlin MoE runner for MiMo-V2 packed MXFP4 experts on SM90 (#42172) 2026-10-01 20:01:03 -07:00
Yuwei An ef867fa40d [Fix][DSV4.1] SWA page size with bounded replay (#42128) 2026-10-01 18:50:43 -07:00
jiaryangandThomas Wang 58f0d250ec [ROCm] Use the fused MLA absorb + RoPE + KV-write kernel for decode-sized forward modes only (#41533)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-01 18:45:29 -07:00
kkandwunhuang 29f6d408c0 [AMD][V4.1][*/N] Fix shared-expert fusion accuracy and speed up MoE routing on ROCm (#42011)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-01 18:19:06 -07:00
Kedar PotdarandClaude Opus 5.5 8acd36b56b [Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX AgentX recipe (#42109)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:11:29 +08:00
Qiaolin Yu 67eab57057 [qwen 3.8 next] Fuse NEXTN verify and draft graph input preparation (#41175) 2026-10-01 17:17:34 -07:00
caihuali95andCheng Wan 2a3655fea1 [unified-memory] Name the fused KV translate for what it computes (7/7) (#40331)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:14:03 -07:00
df530479c6 [unified-memory] Stride the KV translate kernel and route every translate through it (6/7) (#40330)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:12:14 -07:00
9855e9c6f8 [unified-memory] Remove the kernel-page multiplier plumbing (5/7) (#40329)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:11:25 -07:00
4c4e352a4b [unified-memory] Mark the write loc physical and check it at every write door (4/7) (#40328)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:10:37 -07:00
cdadb1c79a [unified-memory] Token-major dense views for the unified memory pool (3/7) (#38592)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:09:38 -07:00
caihuali95andCheng Wan 5df52e6382 [unified-memory] Build paged KV views through one helper (2/7) (#40327)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:08:42 -07:00
caihuali95andCheng Wan d3dbb71e5a [unified-memory] Derive KV row addresses from strides, not shapes (1/7) (#40326)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:07:47 -07:00
shyeh25andBrayden Zhong 5423a4d885 [MegaMoE] Preserve W13 layout for ModelOpt NVFP4 experts (#39388)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-10-01 16:12:34 -07:00
f17f7705a5 [AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 (#39273)
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: long10024070 <long10024070@users.noreply.github.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: sunxxuns <126995791+sunxxuns@users.noreply.github.com>
Co-authored-by: Long Luong <long.luong2+personal@onemount.com>
2026-10-01 15:59:44 -07:00
Jan Bernlöhrandjbernloehr 80bb3fb651 [Fix] Select the MXFP4 MoE runner for MiMo-V2 packed experts on SM100 (#41668)
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com>
2026-10-01 15:33:30 -07:00
Kan WuandKan Wu a21dcbd320 [sgl-router] Support engines started with --api-key (--worker-api-key) (#41996)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-01 14:54:59 -07:00
Kan WuandKan Wu d7565215ec [sgl-router] Route to a specific DP rank inside multi-rank workers (--dp-aware) (#41974)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
2026-10-01 14:50:59 -07:00
Martin Hua 52d5faed5b [Bugfix] Llama4 local attention: read page ids from the graph tables in CUDA-graph capture/replay (page_size > 1) (#38187) 2026-10-01 14:42:22 -07:00
bf2d686a82 Add triton autotune on the Mamba2 SSD kernels (#39130)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-01 14:25:37 -07:00
Jiajun LiandClaude Fable 5.1 dcf4e18283 fix(nccl): disable graph buffer registration for pausable graph pools (#40648)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 12:34:23 -07:00
Shangming CaiandClaude Fable 5 a71c7d19e7 [Router] Document peer bootstrap: flags, RBAC, downward API, probe implications (#42049)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-02 02:20:36 +08:00
3f4bfda13c [Bugfix] Fix DeepSeek V4 Pro TP24 vocabulary-padding failure on Hopper (#31801)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:11:18 -07:00
a0fcebb3bb NIXL: Use stride desc API (#37827)
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-01 10:53:00 -07:00
jain-ria fb92ba1127 [Rust] Wire the gRPC server into the frontend lifecycle (#41964)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-10-01 10:44:56 -07:00
JoeandAlex Nails f03a183719 Fix Kimi-K3 MLA output gate dispatch on non-CUDA devices (#36406)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 08:53:31 -07:00
JoeandAlex Nails 3031091c36 [Kimi-K3] Guard optimized paths by platform (#40269)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 08:53:05 -07:00
Артем Савкин 1bbce57d9e [NPU] [Diffusion] Fix NPU multimodal-gen CI (#41528) 2026-10-01 18:02:37 +03:00
41cbe65de0 [sgl-router] Enforce compatible PD version groups (#41613)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 20:24:20 +08:00
3b2ad1c6ae [sgl-router] Fail fast and cancel decode when prefill fails (#41612)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 19:35:56 +08:00
Aurick QiaoandAurick Qiao 1a0250c5b6 [Disagg] Validate state strides before Mooncake transfers (#41607)
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com>
2026-10-01 18:47:41 +08:00
d669efba84 [AMD][V4.1][*/N] Greedy dspark draft/accept under SGLANG_SIMULATE_ACC_LEN (#41981)
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-01 03:35:28 -07:00
46637965ab [sgl-router] Allow load-only routing without a tokenizer (#41611)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 18:30:35 +08:00
3c4e218779 [AMD][V4.1][*/N] OPUS sparse prefill on gfx950 through layout conversion (#42017)
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: thomawan <thomawan@amd.com>
2026-10-01 03:21:25 -07:00
kkandwunhuang f45c004e18 [AMD][V4.1][*/N] Build DSpark draft metadata inside the CUDA graph on ROCm (#42014)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-01 02:17:13 -07:00
黄孝君 7be0e85658 [CI][NPU][Diffusion] Bump ascend consistency GT to the CANN 9.1.0 baseline (#41500) 2026-10-01 17:13:28 +08:00
Xinyi Song 73ba6513f4 [AMD][V4.1][*/N] Switch the fp8 dense GEMMs on gfx950 to aiter's MXFP8 GEMM (#41970) 2026-10-01 02:04:39 -07:00
Xinyi Song 0224fcbd1c [AMD][V4.1][*/N] Route low-ratio indexer and candidate-block top-k through top-k v2 (#41947) 2026-10-01 01:56:31 -07:00
Cheng Wan 5fab1f0f51 [Fix] Let the aiter DCP ASM decode test run without aiter (#42018) 2026-10-01 16:55:15 +08:00
Thomas Wang 69240ba7d6 [AMD][V4.1][*/N] Pick the gfx950 wo_a batched gemm tile by row count (#41971) 2026-10-01 01:52:39 -07:00
Thomas Wang ffb134188b [AMD] Update aiter version for v41 (#42023) 2026-10-01 01:49:45 -07:00
Shunkangz 266d9d1fa7 [Qwen3.8 CP 1/4] Context parallelism for QSA attention and the sparse indexer (#39721) 2026-10-01 01:28:55 -07:00
b51d4a04f0 [DFlash] Keep grouped-conv taps inside each request's block so NaN/Inf cannot leak across requests (#41736)
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 00:18:37 -07:00
Mohammad Miadh AngkadandMohammad Angkad 42f2e4fcb7 [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer (#41987)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-01 00:16:28 -07:00
DarkSharpness 3ed6367d3f [DSV4/DSA] Name the FlashMLA KV format and drop the V4.1 support probe (#41337) 2026-10-01 00:02:18 -07:00
d010e50ad6 [PPU][1/N] CI: Add backend registration and runner preflight (#39788)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-01 00:01:17 -07:00
amd-danli103 9c3262d855 [AMD][DSV4] feat: enable PD-disagg with fp8 unified_kv on gfx950 (#39166) 2026-09-30 23:42:31 -07:00
315080a0de [sgl-router] Exclude portless prefills and retry bootstrap discovery (#41610)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 14:32:14 +08:00
Shangming CaiandClaude Fable 5 c78fa09290 [Router] Drop the snapshot client's redirect-following fallback (#42010)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:31:53 +08:00
Eric Zhang 947fee9384 [PD] Fix block scale registration for mixed full/SWA KV dtypes (#41924) 2026-10-01 14:25:04 +08:00
Xiaoyu Zhangandkernel-design-agents 77b64e353f [KDA+Kimi K3] Speed up LTX-2.3 QK norm and split RoPE on H200 (#41459)
Co-authored-by: kernel-design-agents <334018530+kernel-design-agents@users.noreply.github.com>
2026-10-01 14:21:31 +08:00
Peng Wu 11500701f1 [DSV4] Reserve the FULL logical page in the c4 indexer pool like its KV pool (#41163) 2026-09-30 23:19:27 -07:00
Shangming CaiandClaude Fable 5 047be72a8c [Router] Harden peer-bootstrap edges: refuse redirects and metric docs (#41999)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:15:42 +08:00
chien-an-chen 99c9d65b32 [AMD] AITER MLA DCP decode ASM path (opt-in) (#40750) 2026-09-30 22:47:55 -07:00
billishyahao dc7defd294 [Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens (#38991) 2026-09-30 22:35:25 -07:00
ashwini rathi 86807b87dd [XPU] Fix test_qsa_indexer_xpu after defer_expansion change (#41915) 2026-09-30 22:34:10 -07:00
Ke Bao 5733bcaad3 Declare HiCache host pools for DSA indexer (#40913) 2026-10-01 13:08:26 +08:00
f02828ea39 fix(dsv4): ship the fp8 unified_kv rope pool through the direct external linker (#40471)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: billishyahao <bill.he@amd.com>
2026-09-30 21:46:56 -07:00
Mohammad Miadh AngkadandMohammad Angkad 40e1bb0f36 [Docs] Add NVIDIA NVFP4 checkpoint to GLM-5.3-Flash cookbook (#41986)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-01 12:15:30 +08:00
3b6d37a56e [Router] Gate readiness on peer bootstrap, and make it observable (13/13) (#40699)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 11:55:46 +08:00
ebdeba2fee [Fix] Emit Responses reasoning content-part events (#41789)
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-01 11:15:24 +08:00
Mohammad Miadh AngkadandMohammad Angkad 99c59a2ea6 Drop the GLM-5.3-Flash breakable prefill chunk-size pin (#41837)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 19:26:58 -07:00
Shuwen Wang bb62f125aa [HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free (#41346) 2026-10-01 10:24:51 +08:00
Shuwen Wang c9d6c966d0 [HiSparse] fix: reject speculative decoding and mixed chunk at startup (#41347) 2026-10-01 10:24:15 +08:00
stanleys12andClaude Opus 5 3fb721917d [Fix] Require admin auth for /load_lora_adapter_from_tensors and /set_trace_level (#41322)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-01 10:23:12 +08:00
Shuwen Wang b62b21e7f4 [HiCache][PD] fix: keep decode load-back polling out of rank-local collectives (#41393) 2026-10-01 10:22:30 +08:00
zijieandZijie Chen 0148d9a2fd [AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm (#41497)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
2026-09-30 19:05:49 -07:00
MickandMick Qian 2ab1b537f8 [diffusion] docs: recommend docker for gpu deployments (#41966)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-01 09:52:31 +08:00
36330b8c08 [Router] k8s e2e for cache-aware peer bootstrap (12/13) (#40698)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-01 09:42:59 +08:00
Shafeeq IbraheemandYanbin Jiang b4df214948 Fix LoRA usage-counter accounting across request lifecycle paths (#31808)
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
2026-09-30 18:37:53 -07:00
Sageandishandhanani baae0195d9 [ci] fix renderer image publish (#41936)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-09-30 18:27:11 -07:00
jain-ria ebccddce19 [Rust] Implement the Rust gRPC adapter for native generation (#41766)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-30 18:17:42 -07:00
Yu-Yun Changandkk 6af651ea0c [AMD][Quark] Serve the Kimi-K3 MXFP4 checkpoint on ROCm (#40811)
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
2026-09-30 18:14:15 -07:00
leonardHONGandlianghougong a702970ecb [Perf] Use FA4 by default for MiMo on SM100 (#41886)
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com>
2026-10-01 09:05:12 +08:00
Sage b37e6f79fc [ci] grant ci permissions to sagearc (#41935)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-30 17:21:41 -07:00
Liangsheng Yin 63321e3693 [Scheduler] Move internal-state readback and updates into a collaborator (#41950) 2026-09-30 16:40:57 -07:00
Liangsheng Yin 517cfa218c [Test] Prune redundant scheduler unit tests (#41952) 2026-09-30 16:30:18 -07:00
Jan Bernlöhrandjbernloehr 488869c2d0 [Fix] Keep MiMo-V2 processor available without TorchCodec (#41667)
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com>
2026-09-30 15:28:16 -07:00
826f15bf9c [Fix] Preserve inline instructions for Responses and Kimi K3 Messages (#40077)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-30 15:20:52 -07:00
Sage 8e4bcc5ca9 [rust-renderer] sglang-processor lib (#41896)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-30 15:09:35 -07:00
f050ab194a fix(spec): enable Qwen3.5 EAGLE3 capture and streaming overlap (#41154)
Co-authored-by: Yuhao Wang <59914901+jin-chuan@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <95566987+hnyls2002@users.noreply.github.com>
2026-09-30 14:41:46 -07:00
Cheng Wan a9c97c9f69 [Feature] Add --attn-dp-size and deprecate --enable-dp-attention (#41818) 2026-09-30 13:34:18 -07:00
Jason Mancusoandhnyls2002 201c4b9b2a [RL, Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl (#34201)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-30 13:32:26 -07:00
Cheng Wan 986cb8268b [Refactor] Enter draft TP scopes by attention ownership and drop ModelRunner.tp_group (#41817) 2026-09-30 13:28:50 -07:00
Cheng Wan e7f6a99313 [Refactor] Add make_pp_layers so models stop handing their PP position down (#41816) 2026-09-30 13:28:34 -07:00
Cheng Wan 00659ac099 [Refactor] Stop passing models' layers the placement they already read (#41815) 2026-09-30 13:28:17 -07:00
Cheng Wan c7fa37a5ca [Refactor] Let FusedMoE's weight-loading helpers read the layer's MoE-TP rank (#41814) 2026-09-30 13:27:58 -07:00
Cheng Wan 499e86db98 [Refactor] Keep only the TP and PP groups on the model runner (#41813) 2026-09-30 13:27:31 -07:00
Cheng Wan a09aedf1ca [Refactor] Drop model placement attributes and parameters nothing reads (#41812) 2026-09-30 13:27:11 -07:00
Cheng Wan b0cc246ff8 [Refactor] Drop placement values nothing reads (#41811) 2026-09-30 13:26:51 -07:00
Cheng Wan 4f3ee94d25 [Fix] Make the Solar model constructible and runnable (#41810) 2026-09-30 13:26:32 -07:00
Cheng Wan a659e6461f [Fix] Size the prefill delayer's gather buffer to its TP group (#41809) 2026-09-30 13:26:12 -07:00
Cheng Wan ecc0f0dec2 [Fix] Pass the draft's attention ownership to DFLASH's eager LiLiCorr scope (#41808) 2026-09-30 13:25:54 -07:00
Cheng Wan e289989933 [Fix] Shard MoE WNA16 and Quark INT4-FP8 weights by the MoE placement (#41807) 2026-09-30 13:25:38 -07:00
Cheng Wan 3f20738846 [Fix] State the configured MoE-DP width in the weight-cache fingerprint (#41806) 2026-09-30 13:25:19 -07:00
Cheng Wan e27b6e0971 [Fix] Read TensorCast's WORLD placement under its current names (#41805) 2026-09-30 13:25:03 -07:00
YAMY 4407a21f3c [Spec][PP] Fix hybrid recurrent-state commit and micro-batch pairing under PP x speculative decoding (#40001) 2026-09-30 13:05:39 -07:00
547286bb5a [Deps] Bump transformers to 5.17.0 (#39012)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-30 12:52:02 -07:00
Mohammad Miadh AngkadandMohammad Angkad 036a3b3e14 [Test] Lower the SM120 NVFP4 KV GSM8K threshold to 0.60 (#41799)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 12:50:42 -07:00
Mohammad Miadh AngkadandMohammad Angkad eae808903f [Deps] Bump FlashInfer to 0.7.0.post1 (#40709)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-30 12:42:19 -07:00
4dfc0ac131 [BugFix] Pass token-major Q/K tensors from Gemma-3 to RadixAttention (#37984)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-09-30 12:05:41 -07:00
Kevin MiandKevin Mi 3a398442bf dsv4.1-amd: serve DeepSeek-V4.1 on gfx950 (#41308)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-30 11:47:14 -07:00
9f9f9e702a [HiCache] Attribute buffer-mode storage hits against the joint device match (#41758)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-09-30 11:46:30 -07:00
zijiexia d26ec4690d docs: remove unreliable DeepWiki badge (#41851) 2026-09-30 10:51:36 -07:00
Shuwen WangandShangming Cai aec4b799e6 [PD][Mamba] fix: free the COW mamba slot of decode requests dropped before preallocation (#41451)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-30 17:50:44 +00:00
jain-ria 21ed9fbf44 [Rust] Extract shared runtime.v1 protobuf bindings (#41726)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-30 10:13:08 -07:00
Sohom chakrabortyandShuwen Wang 28c5e7f5cb [Fix] Stop PD-decode queue_time from counting decode time before a retraction (#41380)
Signed-off-by: Sohom Chakraborty <sohomchakraborty.iitkgp@gmail.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-09-30 23:17:03 +08:00
bd66ce343e [diffusion] nightly: measure every framework with one client end-to-end methodology (#41689)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 19:52:20 +08:00
4b3f63ec70 [Router] Prove a bootstrapped replica answers like the one it copied (11/13) (#40697)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 19:19:26 +08:00
4c930125c7 [Router] Ask the fleet when a graft's splice goes unwitnessed (10/13) (#40696)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 18:59:21 +08:00
Cheng Wan 53225fadf4 [Fix] Stop the namespace census at the leaf a read names (#41838) 2026-09-30 03:27:51 -07:00
f8500474c9 [Router] Share one fleet-wide fetch across a discovery burst (9/13) (#40695)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 18:07:51 +08:00
jacky.cheng 47dcde70f4 [Docs] Add MI355X FP8 agentic recipe to the Qwen3.5 cookbook (#41849) 2026-09-30 17:44:19 +08:00
Yuan Luoandluoyuan.luo 57c81258d4 [KDA] Enable the ptx_kda prefill backend on SM100 (B200 / GB200) (#41445)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-30 17:24:51 +08:00
Yuan Luoandluoyuan.luo c0296186b7 [KDA] Fix ptx_kda prefill NaN without a gate lower bound and workspace growth (#41572)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-30 17:23:54 +08:00
paulzhang-tmandKe Bao 8055ccd2cd [Linear Attention] Expose GDN/KDA prefill hooks and auxiliary cache accounting (#40227)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-09-30 17:19:02 +08:00
EdwardXuyandEdwardXuy 9b7464c8dd ci: temporarily disable A5 (950) nightly job during power maintenance (#41856)
Co-authored-by: EdwardXuy <EdwardXuy@users.noreply.github.com>
2026-09-30 17:17:39 +08:00
jiaryang e9b0d0c0a3 [AMD] Gate flashinfer and TRT-LLM DSA paths on CUDA (#40911) 2026-09-30 02:10:02 -07:00
Michael d7cce54885 [AMD] Resolve QSA packed-varlen decode to aiter on HIP (#41513) 2026-09-30 02:01:15 -07:00
amote-i 964c45cf31 [NPU] [DOC]: add Kimi-K3 NPU PD disaggregation recipes (#41717) 2026-09-30 16:42:46 +08:00
4d4d3f2e5c [Router] Sweep the fleet for a peer snapshot and hand it to the pump (8/13) (#40694)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 16:34:12 +08:00
45c8ddddd3 [diffusion] feat: check the final MP4 in-process instead of spawning ffprobe (#41825)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 16:26:39 +08:00
WenhaoZhang 0931a72eb2 [diffusion] feat: support MiniMax H3 RL (#34365) 2026-09-30 16:23:56 +08:00
Michael b87a241a6f [AMD] Add Qwen3.8-Flash-Next-FP8 nightly validation (#36901) 2026-09-30 00:54:36 -07:00
682f4ef32f [ROCm] Select DSA indexer top-k wave size by arch (wave32 on gfx1250) (#41687)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-09-30 00:47:04 -07:00
Shuwen Wang 51cae5f303 [HiCache][PD] fix: clamp the decode restore to the prefix promised to prefill (#41450) 2026-09-30 15:26:15 +08:00
heziiopandsglang-npu-bot ec0006673a [NPU] Add decode context parallel support for dsa models (#37787)
Co-authored-by: sglang-npu-bot <sglangnpu@163.com>
2026-09-30 15:22:16 +08:00
Shangming Cai 8c75ad9e62 chore: bump mooncake version to 0.3.13.post1 (#41678) 2026-09-30 15:16:54 +08:00
Zhaoyi Li 3b537e96e2 [AMD][DI][CI] Leave the GLM-5.2 MTP decode room to load its Triton kernels (#41135) 2026-09-30 00:02:42 -07:00
Zhaoyi Li 25de118ae3 [AMD][DI][CI] Give the dsv4pro-fp4 MTP decode more memory headroom (#41238) 2026-09-29 23:57:25 -07:00
Zhaoyi Li 4207f1ae23 [AMD][DI][CI] Record the commit the MI355X nightly tested (#41099) 2026-09-29 23:55:38 -07:00
Zhaoyi Li f85ca901e2 [AMD][DI][CI] Fix four failures in the MI355X disaggregation nightly (#41025) 2026-09-29 23:54:40 -07:00
24ee67d2fa Readme refresh (#41787)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-09-29 23:48:38 -07:00
eb9c9ee99d [Qwen4-Exp] Optional fp8 (e4m3) storage for the compressed QSA indexer cache (#39614)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
2026-09-29 22:11:46 -07:00
Qiaolin Yu 88f95f4b87 [qwen next] Fuse QSA KV preparation and sparse block expansion (#40972) 2026-09-29 21:54:34 -07:00
Qiaolin Yu b660100c00 [qwen 3.8 next] Fuse QSA graph replay metadata across draft steps (#41173) 2026-09-29 21:52:28 -07:00
JohnQinAMDandClaude Opus 5.5 964fd6af92 [ROCm] GLM-5.2: gfx950 four-kernel fused DSA indexer decode path (#38583)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:52:00 -07:00
hhhh1252023 be66c1b460 [NPU][CI] Fix stale multimodal-gen job name in fast-fail health check (#41824) 2026-09-30 12:39:23 +08:00
Cheng Wan d41904f05e [Refactor] Build seven more decoders from stage boundaries (#41779) 2026-09-29 21:16:45 -07:00
Cheng Wan 113e6a1496 [Refactor] Simplify layer boundary internals (#41778) 2026-09-29 21:15:58 -07:00
Wenfei (Charles) Qi 7fb85a9d88 [MLX] Fix startup after logical token capacity change (#41314) 2026-09-29 21:15:56 -07:00
Cheng Wan 9ee065e49a [Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends (#41754) 2026-09-29 21:11:37 -07:00
Cheng Wan faa5064f6e [Refactor] Send producer-written residual streams through to_pp (#41753) 2026-09-29 21:11:02 -07:00
Cheng Wan 843f50f5c1 [Fix] Reduce the Sarvam dense MLP through its row-parallel projection (#41752) 2026-09-29 21:10:17 -07:00
Cheng Wan ea855f0daf [Refactor] Leave decoder stacks through final_norm(skip_empty=True) (#41751) 2026-09-29 21:08:19 -07:00
Cheng Wan 01b5f572b7 [Refactor] Rename layer boundary protocols, contracts and helpers (#41750) 2026-09-29 21:05:48 -07:00
cb3ba2f2b7 [XPU] Add dynamic_expert_bias and track_state support (#39618)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-30 12:05:35 +08:00
Alison Shao 8854857a99 [CI] Bound maintenance API requests to 30 seconds (#41800) 2026-09-29 20:02:02 -07:00
Mohammad Miadh AngkadandMohammad Angkad 7b30ff26ee [CI] Evaluate the maintenance gate with github-script instead of jq (#41625)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-29 18:47:51 -07:00
fc9bdc8a3e [MoE] Keep the moe_align_block_size pad fill inside sorted_token_ids (#41759)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: oulgen <oulgen@users.noreply.github.com>
2026-09-29 17:43:17 -07:00
Baizhou Zhang e5cec303ae [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL (#41777) 2026-09-29 17:28:30 -07:00
sglang-bot cae69be584 chore: bump sgl-kernel version to 0.4.8 (#41784) 2026-09-29 17:19:21 -07:00
DarkSharpness 57d4adeb83 [Kernel] Expose FlashMLA kv_format in sgl-kernel sparse decode (#41336) 2026-09-29 17:16:52 -07:00
Jason Mancuso ba3e04c063 [CI] Grant CI permissions to four contributors (#41775) 2026-09-29 16:42:01 -07:00
Cheng Wan 8d2d874758 [Fix] Release consumed residual contributions (#41749) 2026-09-29 16:38:05 -07:00
Jiajun LiandShi Dong 3d4953839c [Fix] MoE: require TopK layer_id to ensure routed expert captures (#40980)
Co-authored-by: Shi Dong <hello@dongshi.me>
2026-09-29 16:29:27 -07:00
Jiajun Li f0e4001930 [Fix] Allocate SWA KV pools inside the memory-saver region (#40979) 2026-09-29 16:29:02 -07:00
Yuwei An d3f8a8f4f5 [Spec/Prefill Coordination] Dspark support (#41623) 2026-09-29 16:08:01 -07:00
Liangsheng Yin 211b1d9784 [Refactor] Share PD routing fields across request models (#41756) 2026-09-29 15:11:18 -07:00
37a47737c8 [Perf] Tune SM90 GDN recurrent verify launch for small batches (#41486)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-29 15:03:33 -07:00
Mohammad Miadh Angkadandhnyls2002 4885b563c3 [Test] Demote PD test RDMA openability check to a warning (#41681)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 14:52:31 -07:00
84523d6785 [Spec] Reuse K3 auxiliary outputs across decode CUDA graph sizes (#40159)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 14:24:51 -07:00
jain-ria 46beacf13d [grpc rust] Define semantic frontend contract (#41709)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-29 13:40:04 -07:00
James LiuandClaude Opus 5.5 77a81996ce [PD] Preserve bootstrap metadata in native Messages requests (#39711)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:32:54 -07:00
3c4653b6a3 [Metrics] Fix PD latency histogram accounting (#39706)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-29 13:12:09 -07:00
zijiexiaandClaude Sonnet 5.5 f884231f5a [Docs] Add IQuestLab card logo to cookbook landing page (#41746)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 12:47:21 -07:00
51ace48387 [Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds) (#40275)
Co-authored-by: Gilford Ting <gilford@modal.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 12:20:22 -07:00
ishandhanani 9a16e47437 chore: grant jain-ria CI permissions (#41728) 2026-09-29 11:56:51 -07:00
ashwini rathi bf80732b92 Fix pip install: exclude multimodal_gen/.claude symlink from package data (#41703) 2026-09-29 11:35:11 -07:00
8f2a9638f7 [metrics] Fix non-streaming TTFT by flushing the first output through detokenization (#38600)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 11:24:52 -07:00
Colin Z 875dd41e6f [AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm (#28734) 2026-09-29 11:20:10 -07:00
Chunan Zeng affc602f51 [CI] Extend DeepGEMM GB300 validation timeout to three hours (#41720) 2026-09-29 10:57:20 -07:00
ced5e9fe49 [Kimi] Enable GB300 TP4 and GB200/GB300 TP16 SP collectives (#35330)
Co-authored-by: Ajit Mistry <ajmistry@amistry.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-09-29 10:18:11 -07:00
CYJiang fa090f7755 [MUSA] Fix fused MoE GEMV registration and torchada pin (#41444) 2026-09-29 09:51:08 -07:00
ashwini rathi 98fce73d5b [XPU] publish nightly docker image with sgl-kernel-xpu built from main (#41650) 2026-09-29 20:09:53 +08:00
4a4e1ce014 [Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13) (#40693)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 20:07:37 +08:00
8d1643b21e [Router] Vet a peer's snapshot before it may touch the tree (6/13) (#40692)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:58:11 +08:00
8b3a4cad8b [Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13) (#40691)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:06:13 +08:00
7bcfcf5aa0 [Router] Watch EndpointSlices for sibling router replicas (4/13) (#40690)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 18:56:32 +08:00
c5a38d3998 [sgl-router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb (#41268)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:55:58 +08:00
MichaelandClaude 5828cd4927 [AMD] Add GLM-5.3 MI30x and MI35x nightly accuracy tests (#41602)
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-29 02:49:01 -07:00
Rumit DesaiandXiaoyu Zhang f731e82f09 Fix MUSA detection under torch.compile fullgraph (#41065)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-29 17:26:27 +08:00
Carrie ChenandBrayden Zhong 0e586fd12d fuse shared experts with routed experts in MegaMoE's DeepGEMM (#39313)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-09-29 17:25:10 +08:00
c9cb32e310 [AMD] fix kda decode flydsl import (#40546)
Co-authored-by: Bingxu Chen <bingxche@amd.com>
Co-authored-by: Michael <13900043+michaelzhang-ai@users.noreply.github.com>
2026-09-29 02:20:42 -07:00
Liang Gong 34a1234d21 chore: expose agent skills via .agents directories (#40515) 2026-09-29 17:12:28 +08:00
Michael 05817a40c9 [AMD] Fix GLM-5.3 quark MoE MI35x test runner config (#41464) 2026-09-29 16:46:52 +08:00
Jiacong Fang bd78095030 [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B (#41059) 2026-09-29 00:38:58 -07:00
MickandMick Qian dc1bd46802 [diffusion] feat: support bounded exact conditioning cache across native models (#40470)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-29 15:27:22 +08:00
eeecho c7be3e935b [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts (#34528) 2026-09-28 23:08:44 -07:00
Anupa SajikumarandMa Mingfei 2ff52e3ceb [Intel][XPU]Enable chunked prefill scnearios for XPU with UT (#33804)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:28:25 +08:00
9ae43b0eee [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path (#40828)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:16:03 +08:00
Cheng Wan 5781206833 [PD] Give FakeKVReceiver ensure_abort_notified (#41618) 2026-09-28 21:59:17 -07:00
jain-riaandishandhanani 069523f999 [Rust] Extract a transport-neutral frontend core (#39385)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-09-28 21:18:50 -07:00
Cheng Wan 6dad152490 [Refactor] Document layer boundary contracts and integration (#41557) 2026-09-28 20:18:27 -07:00
Cheng Wan 49e7bd179d [Refactor] Group layer boundary unit tests (#41556) 2026-09-28 20:18:08 -07:00
Cheng Wan de9f850b36 [Refactor] Rename the module to layer_boundary (#41555) 2026-09-28 20:17:50 -07:00
Cheng Wan 701213c867 [Refactor] Retire the layer facade and simplify boundary internals (#41554) 2026-09-28 20:17:33 -07:00
Cheng Wan 9b89b96230 [Refactor] Migrate specialized decoder and overlap boundaries (#41553) 2026-09-28 20:17:17 -07:00
Cheng Wan 49b5e76e45 [Refactor] Construct independent decoder stage boundaries (#41552) 2026-09-28 20:17:00 -07:00
Cheng Wan acffb0d20d [Refactor] Select reduction fusion at the consumer (#41551) 2026-09-28 20:16:44 -07:00
Cheng Wan cb1fa3c399 [Refactor] Capture auxiliary states at residual reads (#41550) 2026-09-28 20:16:26 -07:00
Cheng Wan 42f58b4da7 [Refactor] Carry residual state across stage boundaries (#41549) 2026-09-28 20:15:03 -07:00
Cheng Wan 6f72cc3b64 [Refactor] Centralize decoder output access (#41548) 2026-09-28 20:14:04 -07:00
Cheng Wan 0a8ca85565 [Refactor] Group communicator fusion and CP adapters (#41547) 2026-09-28 20:10:50 -07:00
Mohammad Miadh AngkadandMohammad Angkad e49d7fb170 Remove GLM-4.1V-9B-Thinking from encoder DP MMMU test (#41608)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-28 19:45:21 -07:00
Liangsheng Yin d9074f7943 [Test] Fail fast when PD test RDMA devices are not openable by ibverbs (#41600) 2026-09-28 19:34:58 -07:00
James 6e1df69623 [npu]support NPU 910C L2 memcache offload (#41527) 2026-09-29 10:30:01 +08:00
77091cea68 [Radix Cache] Sync Rust TreeCore and make it the default (#39627)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: alphabetc1 <alphabetc1@users.noreply.github.com>
2026-09-29 10:18:57 +08:00
WenhaoZhang 49d26ff0c2 [diffusion] model: support flux 3 action robot policies (#41066) 2026-09-29 10:17:19 +08:00
79cafec013 [Model] Add IQuest Q1 support and MTP draft (#41590)
Co-authored-by: zelong huang <yzhu@ubiquant.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: zelong518 <zelonghuang05@gmail.com>
Co-authored-by: chenwenxiaolive <cocoshirolive@gmail.com>
2026-09-28 17:21:31 -07:00
Stanislav PetrovandStanislav Petrov b95329a74d [Docs] Add GigaChat 3.5 and GigaChat 3.5 Reasoning cookbook pages (#41118)
Co-authored-by: Stanislav Petrov <stapetrov@sberbank.ru>
2026-09-29 08:16:22 +08:00
7889fbab9a [PD] Keep EAGLE DP graph and token metadata consistent (#32196)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-28 17:15:51 -07:00
Shuwen Wang 89e1316eae [mem cache] refactor: remove the index-K continuous getters orphaned by the CP v1 removal (#41469) 2026-09-28 16:41:07 -07:00
Hrithvik AlexandHrithvik Alex f124c5ceec [Model Loader] Stop checkpoint prefetch after iterator completion (#41588)
Co-authored-by: Hrithvik Alex <hrithvik@baseten.co>
2026-09-28 16:27:09 -07:00
9654e5c4bc dsv4.1-amd: fused mHC boundary and all-reduce + mHC post kernels (#41021)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-28 16:15:08 -07:00
Kevin MiandClaude Opus 5.5 096b066fb4 dsv4.1-amd: gfx950 sparse decode attention and sorted top-k (#41020)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 16:14:28 -07:00
Zhang, Jiejing 26d7539799 [Docs][AMD] Update GLM-5.2 MI355X daily image (#41597) 2026-09-28 15:42:54 -07:00
Trevor Morris b07cb9e9e7 [NVIDIA] Update deepgemm, deep-ep, sgl-kernel in CUDA 13.4 image, use cuda base image (#40987) 2026-09-28 14:12:46 -07:00
Nan Jiang 6fa3fe69e2 [PD] Keep the sampling mask of a replayed rebootstrap token (#41235) 2026-09-28 12:02:27 -07:00
Shangming CaiandClaude Fable 5 2a1c477648 [PD] Defer decode KV release on every transfer failure, not only decode-initiated aborts (#41404)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-29 02:19:51 +08:00
YAMY f209e67767 [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism (#39643) 2026-09-28 10:22:40 -07:00
Kurkur 6624999385 [Fix][NPU] fix dp-attn hang when pin_mem is True on NPU (#40446) 2026-09-29 00:41:43 +08:00
triple-mu 2e7e0802f4 [diffusion] Qwen-Image 2.1: fuse Q/K RMSNorm + RoPE + KV packing into one CUDA kernel and project Q/K/V with one packed GEMM (#41339) 2026-09-28 21:45:39 +08:00
DarkSharpnessandClaude Opus 5.5 8151378c1a [DeepSelect] Add page-table transform to top-k and tighten the layout contract (#41364)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 21:43:57 +08:00
DarkSharpnessandClaude Opus 5.5 0318a8d0af [Doc] Add kernel benchmark rule on L2 cache reuse (#41545)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:25:58 +08:00
Shuwen Wang da3eb6db32 [Unified Memory] Fix Inkling conv-checkpoint track ids written to virtual slot numbers (#41144) 2026-09-28 05:19:19 -07:00
Shuwen Wang bddeb61d94 [Unified Memory] fix: preserve FP8 dtype in unified MHA pool (#38133) 2026-09-28 05:19:03 -07:00
MickandMick Qian 090439bf8f [diffusion] docs: consolidate Qwen-Image 2.1 guidance in its cookbook (#41540)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 20:04:32 +08:00
minke.yu e403cede0a deploy: D mr320/cg256/mf0.83 (KV pool 41.8M tok/worker); d1 HTTP 30001->30030 fix port collision with p1 scheduler loopback bind 2026-09-28 20:00:11 +08:00
6aacca2d91 [Router] Name a replica's siblings with --kv-peer-selector (3/13) (#40689)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:49:09 +08:00
Raiden MakotoandRaiden-Makoto fa443412c5 [AMD] [GLM5] Fuse shared expert into AITER MoE on gfx950 (#41161)
Signed-off-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
2026-09-28 03:40:09 -07:00
2f88c65289 [Router] Serve the cache-aware tree at /internal/kv_snapshot (2/13) (#40688)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:38:52 +08:00
47ad1928a2 [sgl-router] Scope input_ids forwarding by renderer: all text chats for DeepSeek-V4 (#41226)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-28 18:37:40 +08:00
karverma-amd 85be04978d [AMD][DSV4] moe: enable shared-expert fusion on the grouped-topk path (megamoe) (#40943) 2026-09-28 03:21:57 -07:00
Denji-kk 6caf0ff4ae perf(multimodal): offload CPU feature hashing with bounded admission (#39539) 2026-09-28 17:51:05 +08:00
MickandMick Qian c02eaadeb2 [diffusion] optimization: populate cpu weight stores before host registration to speedup layerwise-offload initialization (#40439)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 17:45:58 +08:00
Peng Wu 6582425854 Let predicate-registered linear-attention models carry the mamba radix-cache leaves (#41165) 2026-09-28 17:07:35 +08:00
df378bca42 [Router] Give the cache-aware tree a snapshot surface (1/13) (#40687)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 16:59:59 +08:00
minke.yu 735434fcde deploy: k8s RBG 3p1d cw manifest; D cg512 + load-balance-method total_requests 2026-09-28 16:47:16 +08:00
Peng Wu 546bfa7221 [Kimi-K3] Merge fused_qkvg_proj into the loader-seeded packed_modules_mapping (#41164) 2026-09-28 16:46:11 +08:00
ashwini rathiandPramod Kumar e8eeff2956 [XPU] Bump sglang-kernel-xpu wheel to v0.3.0 (#41220)
Co-authored-by: Pramod Kumar <144990617+pramodkumar-habanalabs@users.noreply.github.com>
2026-09-28 16:15:12 +08:00
Baizhou Zhang 7ee7bef79d docs: add prefill context parallelism guide and design draft (#39354) 2026-09-28 00:26:02 -07:00
Ke Bao 6405cf2881 Fix chat template cache key order (#41517) 2026-09-28 15:17:09 +08:00
flb_ f4de6abee6 feat(npu): Support returning indexer top-k results (#39060) 2026-09-28 14:46:45 +08:00
Rain Jiang 6c4f36e47a Rust server unify datapath for mm and generate requests (#39679) 2026-09-27 23:36:07 -07:00
e75e3b8a98 [Bugfix] fix(hicache): wait for decode offload before retraction (#30899)
Co-authored-by: agent <agent@local>
Co-authored-by: Kevin Flansburg <6134007+kflansburg@users.noreply.github.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-27 23:24:19 -07:00
6be9c78cf4 [Intel GPU] Xpu/weekly simple model enablement 2026 09 21 (#40664)
Co-authored-by: Amrutha M <amrutha.m@intel.com>
Co-authored-by: charlesxu91 <charlesxu.mi@gmail.com>
Co-authored-by: KMS07 <meher.sai.kotthagattu@intel.com>
Co-authored-by: Avi Fenesh <aviarchi1994@gmail.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-28 13:49:18 +08:00
34f090053e [AMD] [Docker] Remove unused LLVM 18 setup from ROCm TileLang build (#41387)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
2026-09-27 22:41:08 -07:00
25de80780a [AMD] Add GLM-5.3-Flash MI35x nightly test (#36903)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 22:40:03 -07:00
Michael 3d2827236d [AMD] Register Triton data movement tests in PR CI (#41137) 2026-09-27 22:38:28 -07:00
Khoa Pham 55cc90b533 [CI] Real-model Kimi-Linear PD parity at page, DCP virtual-page, chunk and cached-prefix boundaries (#41378) 2026-09-27 20:23:07 -07:00
Xinguo Zhu cc012abd21 [CI] fix CI regression on xeon (#41002) 2026-09-28 10:28:01 +08:00
Yihao WangandMick Qian 8b2ca8ecc2 [diffusion] feat: support multiple task types for pipelines (#38762)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-28 10:00:45 +08:00
Kurkur 6cbdfec453 [Fix][NPU] Fix performance degradation caused by serial execution of two-stage ACL op calls on graph (#40814) 2026-09-28 09:32:16 +08:00
ashwini rathi 0823651b0b [XPU] Disable test_ngram_corpus on XPU and detect XPU tests dynamically in CI filter (#41075) 2026-09-28 09:09:40 +08:00
Cheng Wan 81f27fb3a7 [Refactor] Replace LayerScatterModes with LayerFacts and remove ScatterMode (#41443) 2026-09-27 16:56:11 -07:00
Cheng Wan 06d012eaca [Refactor] Give a layer its CuTe DSL kernels at construction instead of a subclass (#41442) 2026-09-27 16:55:47 -07:00
Cheng Wan 341d5273bd [Refactor] Build the boundary into any stage with one construction (#41441) 2026-09-27 16:55:25 -07:00
Cheng Wan 644014bc98 [Refactor] Declare each stage's residual read and update, and give each stage its own entry (#41440) 2026-09-27 16:55:04 -07:00
Cheng Wan 0e907b755d [Refactor] Split the layer communicator into a package (move only) (#41439) 2026-09-27 16:54:41 -07:00
Cheng Wan 427c9e05c1 [Refactor] Choose every layer's boundaries from declarations and remove the scatter-mode selection (#41438) 2026-09-27 16:54:18 -07:00
Cheng Wan 55d2ee5546 [Refactor] Step-3.5: complete the dense MLP's sum through ffn_exit (#41437) 2026-09-27 16:53:57 -07:00
Cheng Wan 0971450c16 [Fix] LongCat-Flash under attention DP: branch and merge the dense FFNs through the communicators (#41436) 2026-09-27 16:53:36 -07:00
Cheng Wan 4fac02fe5d [Refactor] Choose MoE layers' boundaries from declarations when moe_dp_size equals attn_cp_size (#41435) 2026-09-27 16:53:15 -07:00
Cheng Wan ecb8e9fc8f [Refactor] Falcon-H1: complete the FFN's sum through ffn_exit (#41434) 2026-09-27 16:52:53 -07:00
Cheng Wan 19eb56fa57 [Fix] Falcon-H1: count the Mamba mixer's output once under tensor parallelism (#41433) 2026-09-27 16:52:31 -07:00
Cheng Wan 31329c222c [Fix] Run MoE layers under attention DP and GQA prefill CP on the declared DP × CP gather (#41432) 2026-09-27 16:52:10 -07:00
Cheng Wan f0ecf15c84 [Refactor] Move CuTe DSL-fused layers onto the declared boundaries and FFN-exit kernel entries (#41431) 2026-09-27 16:51:48 -07:00
Cheng Wan 8c43c667cb [Refactor] Nemotron-H: build each layer's boundaries from its stage and the previous one (#41430) 2026-09-27 16:51:27 -07:00
Cheng Wan 6f370c3bcd [Refactor] Build each decoder boundary from the declarations of its two sides (#41429) 2026-09-27 16:51:05 -07:00
Cheng Wan 2e7f0bb28c [Refactor] Publish the LoRA token layout from the batch's FFN input rows (#41428) 2026-09-27 16:50:43 -07:00
Cheng Wan c289656434 [Refactor] Let the MoE declare whether its skipped reduction is one TP all-reduce (#41427) 2026-09-27 16:50:23 -07:00
Cheng Wan deaa7f1463 [Refactor] Run two-batch-overlap layers on the declared boundaries (#41426) 2026-09-27 16:50:03 -07:00
Cheng Wan 6f2d2437b6 [Refactor] Run MHC layers on the shared boundary steps with MHC's residual operations (#41425) 2026-09-27 16:49:42 -07:00
Cheng Wan f5ae989a0e [Refactor] Choose fully-DP dense and DSA / MLA prefill CP layers' steps from declarations (#41424) 2026-09-27 16:49:21 -07:00
Cheng Wan cad9eaf27d [Fix] Gather a dense FFN's input across attention DP and CP in one DP sum (#41423) 2026-09-27 16:48:59 -07:00
Cheng Wan 1c919e401d [Fix] Keep one copy of CP-replicated rows in the DP gather (#41422) 2026-09-27 16:48:39 -07:00
Cheng Wan 7a833ad0a2 [Refactor] Choose a GQA prefill CP extend's steps from declarations (#41421) 2026-09-27 16:48:17 -07:00
Cheng Wan 380a4d0332 [Refactor] Run every batch of a layer from one BoundarySteps (#41420) 2026-09-27 16:47:55 -07:00
Cheng Wan 9f87d28db6 [Refactor] Choose the LayerNorm SP region's and input-scattered batches' steps from declarations (#41419) 2026-09-27 16:47:32 -07:00
Cheng Wan 846181f1c4 [Refactor] Give the fused prepare_mlp kernels an explicit contract (#41418) 2026-09-27 16:47:11 -07:00
Cheng Wan 446d181e23 [Refactor] Choose the boundaries of plain-TP dense layers and MoE layers from declarations (#41417) 2026-09-27 16:46:50 -07:00
Yoni GozlanandYoni Gozlan 583fada954 Port chat_parsing core (#40477)
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
2026-09-27 16:14:24 -07:00
78eee88113 [Spec] Add LiLiCorr: a candidate-lattice reranker for DFlash drafts (#37462)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 15:36:48 -07:00
Shangming CaiandClaude Fable 5 14ab74ef95 [PD] Fan drain abort ACKs out to every decode peer of the room (#41402)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-28 05:16:23 +08:00
Thomas Wang b252aceffe [AMD] Update v4 cookbook for megamoe, fp8 kv attn, BCG (#41458) 2026-09-27 10:53:21 -07:00
e581520c67 [HiCache] Add the page-unified KV load-back JIT kernel (#39726)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-28 01:45:02 +08:00
Xinyuan Tong d10a09ecf3 Add MiniMax arch fallback to auto parser resolution (#40930) 2026-09-28 01:33:27 +08:00
CuzMi b2800461e5 [Diffusion] Reuse bit-exact packed SwiGLU for Ming-Image (#41266) 2026-09-27 23:57:43 +08:00
1ba84a4b59 [bench] Take each request's prompt length from the server (#39889)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-27 23:57:18 +08:00
Yuan Luoandluoyuan.luo c311bc9611 [VLM] Introduce FA4 into ViT for SM100/SM103 (#41344)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-09-27 23:04:21 +08:00
588049fe2e [diffusion] fix: run an all-valid attention mask on the backend's unmasked kernel (#41309)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 22:54:46 +08:00
Kangrui Du 84622ce9d5 [diffusion] fix: lora-wrapped linears crash qwen-image and minimax-h3 inference (#41272)
Signed-off-by: rockdu <kangrdu@gmail.com>
2026-09-27 22:54:10 +08:00
a0ba1968b1 [Diffusion] Apply latent-ids and packing to caller-provided initial latents (#34418)
Co-authored-by: aimicahchen <aimicahchen@tencent.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
2026-09-27 18:32:20 +08:00
WenhaoZhang f1e62e3a2e [diffusion] feat: add MiniMax-H3 to ComfyUI integrated mode (#35990) 2026-09-27 17:58:54 +08:00
WenhaoZhang ae47bcd4da [diffusion] feat: minimax-h3 spectrum skip-step + fused RMSNorm/AdaLN (#35684) 2026-09-27 17:56:13 +08:00
WenhaoZhang 2a14f65fd7 [diffusion] feat: support in-place lora merge/unmerge under layerwise offload (#36192) 2026-09-27 17:54:40 +08:00
Liangsheng Yin 37556c1b91 [mem_cache] Replace cache_finished_req with insert_req; release_kv_cache frees and unpins (#41281) 2026-09-27 02:53:16 -07:00
Xiaoyu Zhangandkernel-design-agents 0bc074db83 [KDA+Kimi K3] Speed up SANA-Video residual gate add on H200 (#41305)
Co-authored-by: kernel-design-agents <334018530+kernel-design-agents@users.noreply.github.com>
2026-09-27 16:57:12 +08:00
Qiaolin Yu aa7a976807 [qwen 3.8 next] Fuse small CUDA graph input buffer copies (#41166) 2026-09-27 01:27:56 -07:00
minke.yu e17fbc9a2d deploy: 3p1d-otlp cs32k variant (chunked_prefill 32768 on P)
A/B vs dockerserve-3p1d-otlp.yml (chunk 16384), same image dsv41-pd-visioncp-db7d2cb7d-fix:
- bs1 160k prefill 61950 -> 66084 tok/s (+6.7%)
- bs16 4k prefill 47725 -> 70463 tok/s (+47.6%, fewer scheduler rounds)
- decode TPOT within dspark noise; trace: 160k sender 10->5 chunks, 3.16s->2.98s
Bench logs: bench_cs32k-20260927.log (round2) vs bench_otlptrace2-20260927.log
2026-09-27 14:31:33 +08:00
ShunkangzandZhangheng df1fc6ab64 [kv-shard 3/4] Enable Control Plane B (#39964)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-27 14:19:08 +08:00
Jianghaiandaimicahchen acc15c193b [Diffusion] Preserve per-sample rollout trajectories across multi-output merge (#34416)
Co-authored-by: aimicahchen <aimicahchen@tencent.com>
2026-09-27 13:08:45 +08:00
Shuwen Wang 38d865489a [DSV4.1][HiCache] fix: wait for the layer transfer before reading low-ratio index-K (#41345) 2026-09-26 22:04:03 -07:00
Liangsheng Yin d27efca353 [mem_cache] Never free the protected prefix on request release (#41312) 2026-09-26 22:00:02 -07:00
minke.yu 5507fff133 deploy: add 3p1d soak/prof compose + otlp tracing variant (jaeger badger 6h) 2026-09-27 12:10:00 +08:00
b4584ab838 [sgl-router] Take SGLang's render defaults: --default-chat-template-kwargs and thinking/effort envs (#41221)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 11:01:17 +08:00
Kevin Miandx effb752188 dsv4.1-amd: KV cache layouts, FP4 indexer, compressor and router kernels (#41019)
Co-authored-by: x <x>
2026-09-26 19:44:47 -07:00
Ankith Averineni 425a1f8f24 [AMD] Honor an explicit triton moe_runner_backend for mxfp8 on ROCm (#41377) 2026-09-26 17:18:08 -07:00
Lianmin ZhengandLu Fang fc9e1c8d29 [MemCache] Fix LMCache component cursors and per-cache backend selection (#41328)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-09-26 11:24:39 -07:00
c10fa03fc4 Make sliding-window caching and speculative batch padding extensible (#41325)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: jasonjk-park <jasonjk@fb.com>
2026-09-26 11:24:02 -07:00
Thomas Wang c73f7077eb [AMD] Fix jit broken on rocm env (#41356) 2026-09-26 09:59:59 -07:00
Khoa Pham 7bdd8fe6ec [CI] Merge the Kimi-Linear PD DCP4 nightly tests and drop exact-token parity (#41321) 2026-09-26 09:48:00 -07:00
ArielandClaude Fable 5.1 e1e97e8b49 Pin triton_kernels num_warps for MXFP4 MoE below Hopper (6x gpt-oss decode on RTX 4090) (#41292)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 00:43:25 +08:00
Mohammad Miadh AngkadandMohammad Angkad cdbea5dccf [CI] Move DeepSelect into the attention kernel group to fix the namespace test (#41349)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-26 21:39:00 +08:00
Shangming CaiandClaude Fable 5 710a41ba57 [sgl-router] Book the input_ids forwarding outcome only for built bodies (#41342)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-26 19:39:01 +08:00
DarkSharpnessandClaude Opus 5.5 a947b56bc7 [CI] Add __main__ entry to test_deep_select.py (#41341)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 18:12:41 +08:00
Cheng Wan bd37e7f513 [Refactor] Choose a dense layer's boundaries under attention DP from both sides' declarations (#41257) 2026-09-26 02:41:43 -07:00
Cheng Wan 9cd6616c24 [Refactor] Pick the two-batch-overlap split's layout moves once and remove execute (#41256) 2026-09-26 02:41:09 -07:00
Cheng Wan d3fcf8997c [Refactor] Run the LayerNorm SP region's boundary steps in the communicator itself (#41255) 2026-09-26 02:40:34 -07:00
Cheng Wan 35b369ebba [Refactor] Declare at construction the layers whose FFN completes its own reduction (#41254) 2026-09-26 02:39:58 -07:00
Cheng Wan c081d4aacc [Refactor] Drive an FFN exit's flags and its completion from one selection (#41253) 2026-09-26 02:39:25 -07:00
Cheng Wan 8fc3ce48da [Refactor] Choose prepare_attn / prepare_mlp steps and fused kernels at construction (#41252) 2026-09-26 02:38:03 -07:00
7bc988446d [DeepSeek V4.1] Add DeepSelect JIT kernel. (#40556)
Co-authored-by: DarkSharpness <2040703891@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 17:36:48 +08:00
Hexu Zhaoandhexu.zhao a5c8afab48 [Perf] Mamba2 selective_state_update up to 2x faster on B200 via 8x1 launch config for dstate 128 (+7.5% Nemotron-3-Super serving) (#41223)
Co-authored-by: hexu.zhao <zhaohexu2001@gmail.com>
2026-09-25 23:27:01 -10:00
Sundara Raman Ramachandran 3ed56a341b [Score API] Setwise scoring: CausalLM support (batched + --enable-mis) (#41188) 2026-09-25 23:02:44 -07:00
Mohammad Miadh AngkadandMohammad Angkad 4bb645b8e2 [DSA] Fix the pooled-indexer breakable prefill bridge under DP attention (#41311)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-25 22:52:22 -07:00
8772916e06 [DSv4.1] Move the low-ratio index top-k into dsv4/low_ratio_indexer (#41125)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-25 22:22:57 -07:00
7152aabdcb [sgl-router] Track input_ids forwarding outcomes per chat request (#41185)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 13:09:46 +08:00
Xiaoyu Zhang 38ec649048 [Refactor] Restore logical kernel groups and test organization (#41243) 2026-09-26 12:30:44 +08:00
MickandMick Qian e538489e0e [diffusion] fix: copy small files into overlay materialized trees instead of linking them (#41298)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-26 12:22:47 +08:00
WMCandjacky.cheng dfdbff931d [diffusion] fix: fix Qwen-Image 2.1 default RGBA output (#41150)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-26 12:21:24 +08:00
Lianmin ZhengandLu Fang 3ecb6cd729 [MemCache] Unify component eviction cursors and lock receipts (#41276)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-09-25 20:53:35 -07:00
Olga Miroshnichenko 0e2aac500b [ROCm][Perf] aiter: page-level KV view for gfx950 fp8 page-64 asm prefill (#36505) 2026-09-25 19:23:46 -07:00
Liangsheng Yin f35ec18d8b [Test] Remove more unit tests that mirror implementation or never run in CI (#41297) 2026-09-25 19:22:14 -07:00
Shuwen Wang 4273e7de77 [PD] fix: cache resumed decode-radix requests from root instead of an unlocked re-match (#41261) 2026-09-26 10:21:41 +08:00
Mohammad Miadh AngkadandMohammad Angkad 63d320c723 [DSA] Chunk the kpool indexer MQA logits by query rows under a free-memory budget (#40854)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-25 19:13:13 -07:00
MickandMick Qian 2ee82fe158 [diffusion] model: support Anima Base v1.0 (#41011)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-26 09:53:51 +08:00
Liangsheng YinandDarkSharpness 0967a013c2 [DSv4.1] Move the ratio-1/2 index top-k ops into kernels/ops/attention/dsv4 (#41291)
Co-authored-by: DarkSharpness <2040703891@qq.com>
2026-09-25 18:28:04 -07:00
Xiaoyu Zhang 85374532f2 [diffusion] Fuse lossless Klein packed QK RMSNorm and RoPE on Hopper (#40490) 2026-09-26 09:24:21 +08:00
Liangsheng Yin c842d32b0a [CI] Harden /rerun-test dispatch and partitioning (#41285) 2026-09-25 17:39:50 -07:00
50e50e13ab [DCP] Use logical token capacity for PD admission and load reporting (#39731)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-25 17:39:47 -07:00
Liangsheng Yin 1f6ce4b068 [Test] Remove unit tests that only mirror implementation or never run in CI (#41286) 2026-09-25 16:15:02 -07:00
Liangsheng Yin 2afd343934 [misc] Call all_gather_single / reduce_scatter_single to drop torch deprecation warnings (#41284) 2026-09-25 16:14:13 -07:00
Liangsheng YinandBBuf efd9a40bb8 [PD] Share one head-slice helper across mooncake, mori, and nixl (#39660)
Co-authored-by: BBuf <1182563586@qq.com>
2026-09-25 16:13:18 -07:00
Yuwei AnandClaude Opus 5.5 95047d3464 [Fix] Fix cpu CI fail introduced by pr #38652 (#41287)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 15:55:21 -07:00
Liangsheng Yin 516ab77619 [Test] Route all sgl-eval benchmarks through run_sgl_eval and deprecate run_eval (#41280) 2026-09-25 15:48:38 -07:00
Kevin MiandClaude Opus 5.5 27e883a20d dsv4.1-amd: gfx950 MXFP8 matmul kernels and fp8-grid producers (#41018)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 15:45:40 -07:00
Erik WijmansandEthan Su c421d16563 [sglang][lora] Support DP attention in LoRA backends (#36389)
Co-authored-by: Ethan (Yusheng) Su <yushengsu@radixark.ai>
2026-09-25 12:03:47 -10:00
cf5df82680 [KVCache] Support lmcache unified radix cache (#38652)
Signed-off-by: chunxiaozheng <1179548172@qq.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:08:45 -07:00
Ethan (Yusheng) Su d3e3945e41 [LoRA] Size dense row/column-parallel LoRA buffers from the base linear's real shard (#39379) 2026-09-25 14:01:08 -07:00
Liangsheng Yin f88572a57c [Test] Add in-process sgl-eval adapter and move validated GSM8K tests to it (#41216) 2026-09-25 13:59:17 -07:00
Liangsheng Yin f337f01b80 [Test] Remove dead eval modules and point GSM8K/MMLU docs to sgl-eval (#41215) 2026-09-25 12:50:31 -07:00
ac62ddf164 [Rust frontend] Decode input_ids without untagged buffering (#41246)
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Co-authored-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
2026-09-25 12:47:38 -07:00
Nan Jiang 24b6930076 [Sampling] Stream sampling masks as per-request arrays (#40986) 2026-09-25 12:20:25 -07:00
6d35cee7fc Fix multimodal feature offload races (#40621)
Co-authored-by: cctry <cctry@fb.com>
Co-authored-by: Yongji Wu <yongji@meta.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-09-25 12:18:08 -07:00
metamergebotandYonghao Zhuang 671000aee6 [unified-memory] Honor move gates in float relocation and size auto HiCache from host capacity (#41248)
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
2026-09-25 12:17:28 -07:00
31b931edc3 fix(multimodal): return 400 for corrupt image inputs (#28131)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Justin Tong <justintong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-09-25 19:51:33 +02:00
0154f72b48 [ROCm][Bugfix] Keep quantization for mixed Quark Qwen3.5 MTP checkpoints (#39064)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-25 10:31:29 -07:00
Even Zhou 67bb6a58d0 Revert "[NPU] Fuse FIA KV-cache K/V writes into one npu_scatter_pa_kv_cache call" (#41132) 2026-09-25 23:56:22 +08:00
Xiaoyu Zhang 2f5c9ac43d [diffusion] Enable lossless Cosmos3 Super T2I QK fusion on Hopper TP2 (#40486) 2026-09-25 18:01:06 +08:00
Jianghaiandmickqian a749d84303 [Diffusion] Fix AttributeError in grouped forward_batch by installing the residency manager (#34417)
Co-authored-by: mickqian <mickqian@users.noreply.github.com>
2026-09-25 17:27:38 +08:00
ashwini rathi 62122c838a [XPU] Disable test_ngram_corpus on XPU and extend XPU CI path filter (#41224) 2026-09-25 01:54:05 -07:00
Cheng Wan 3450d68d4e [Refactor] Decide an FFN exit's completion once and declare the group it owes (#41200) 2026-09-25 01:50:35 -07:00
Cheng Wan 26a3214cda [Refactor] Take a layer's last-layer fact from its scatter-mode plan (#41199) 2026-09-25 01:49:33 -07:00
Cheng Wan 8f5a636f8d [Refactor] Build prepare_mlp and the layout moves from named steps (#41191) 2026-09-25 01:48:30 -07:00
Cheng Wan b7f6d04a9a [Refactor] Move Step-3.5, GLM5-Next, Dots3, MiniMax-M3 and Qwen3.5 onto ffn_exit (#41198) 2026-09-25 01:47:43 -07:00
Cheng Wan 963e9fb42d [Refactor] Leave the FFN reduction to the next layer under attention DP (#41197) 2026-09-25 01:47:03 -07:00
Cheng Wan 9d7f44bdbb [Refactor] Carry a deferred FFN all-reduce as UnreducedOutput and complete it in the next layer without the fused kernel (#41196) 2026-09-25 01:46:03 -07:00
Cheng Wan 402df23188 [Fix] Stop counting a deferred FFN sum more than once: replicated TP1 shared expert, dense reduce_scatterv (#41195) 2026-09-25 01:45:00 -07:00
Cheng Wan 1409f469be [Fix] Plan NextN / MTP draft layers as one-layer models and fix the Bailing V2 NextN draft (#41194) 2026-09-25 01:43:55 -07:00
Cheng Wan 51c92e8df1 [Fix] Complete the all-reduce when the flashinfer fused norm declines a batch (#41193) 2026-09-25 01:42:16 -07:00
kk 515f5be77e [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4 (#35619) 2026-09-25 00:47:08 -07:00
Shuwen Wang 434c2e3adc [HiCache] fix: Drain pending backups before internal Mamba write-back (#41092) 2026-09-25 15:06:24 +08:00
Shuwen Wang cbe137727e [HiCache] Give trailing sidecar storage transfers a contiguous prefix_keys chain (#40456) 2026-09-25 15:06:05 +08:00
MickandMick Qian 0fb699ac03 [diffusion] model: support Ming-Image Design and Design-Layer (#41067)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-25 14:44:37 +08:00
metamergebotandXingyu Liu 9c8340a2e5 [DeepEP v2] Let a model package supply its per-rank prefill dispatch bound (#41201)
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
2026-09-24 23:39:03 -07:00
Xinyi Song ec070ec8c8 [AMD] Fix int32 offset overflow in Triton DSv4 KV store kernels (#41159) 2026-09-24 23:11:59 -07:00
Ting SUNandXinyuan Tong 3e7e652900 fix(openai): reject request-supplied chat_template by default (#28135)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-09-25 07:48:34 +02:00
Richard Wang 16d1c93b3d [feat] add a system one compatible /v1/systemone route (#41208) 2026-09-25 12:59:11 +08:00
cec70a4708 [Fix] Keep the target's DP sync slot in draft scopes (#41062)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
2026-09-24 21:48:39 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3fc7a669bf [CI] Move GLM-5.2 layer-split test to extra-b-test-8-gpu-b300 (#41207)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-24 21:25:48 -07:00
e0b4d66733 [Perf] Lazy-load built-in model definitions and nixl_ep at startup (#41061)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
2026-09-24 20:29:33 -07:00
7325b38412 [PD] Honor gracefully_exit in disaggregation event loops and keep non-zero-rank launchers alive on SIGTERM (#40793)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Shiyan Deng <dsy842974287@meta.com>
2026-09-24 20:29:07 -07:00
akhilg-nv 37ebcac3c4 Support XQA backend for SpecDec verify (#32269) 2026-09-24 20:14:46 -07:00
MickandMick Qian 6103a8c743 [diffusion] feat: add opt-in SRT prompt enhancement to image and video APIs (#41095)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-25 10:00:04 +08:00
jacky.cheng 9dc4c5d891 [AMD] Restore non-DCP Mamba checkpoint donation to fix agent-mode cache hit at high conc with HiCache (#40907) 2026-09-24 18:08:10 -07:00
Liangsheng Yin 8ca82118e0 [mem_cache] Drop is_insert from cache_finished_req; release rows from release_kv_cache (#40988) 2026-09-24 17:23:26 -07:00
Shuwen Wangandhnyls2002 a2025b8c4d [DSV4] Account for FlashMLA physical KV page padding in memory budgets (#41091)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-24 17:14:08 -07:00
Sundara Raman Ramachandran 2f2f9d12f8 [Score API] Setwise Scoring Support (#38965) 2026-09-24 16:50:58 -07:00
7c5bdb940c [PD] Add a none decode retraction backup and subclass seams in the PD queues (#41103)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <zqx@meta.com>
2026-09-24 16:13:54 -07:00
466e985f4b [HiCache] Batch buffer-only KV backups within each flush (#40960)
Co-authored-by: oss-sync bot (pranjalssh) <pranjalssh@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-24 16:13:02 -07:00
Caio RochaandCaio Rocha e047e50d4e Add 8-node AllReduce/AllGather and MNVLS algorithm support to MSCCL++ (#37442)
Co-authored-by: Caio Rocha <caiorocha@microsft.com>
2026-09-24 15:51:29 -07:00
Yuwei An 3ec86307f2 Fix mixed chunk prefill with DP speculative coordination (#41179) 2026-09-24 15:41:49 -07:00
6bbd689ab4 [Bugfix] Align DeepSeek-V4.1 reasoning effort budgets (#39929)
Co-authored-by: fengtinglei <fengtinglei@bytedance.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-24 15:21:06 -07:00
Liangsheng Yin 0c578d97fe [DSV4] Size compressed pools from one per-ratio table in DSV4PoolConfigurator (#41049) 2026-09-24 15:19:02 -07:00
Qiaolin Yu a1eb691ab5 [Docs] Enable Qwen3.8 Flash Next NVIDIA NVFP4 on B200/B300/GB300 (#41046) 2026-09-24 14:19:00 -07:00
Liangsheng Yin 182f62d2de [Fix] Patch set_dp_buffer_len_from_batch in DP spec prefill coordination test (#41162) 2026-09-24 13:52:11 -07:00
Shuwen Wang 961404b010 [DSV4] Fix TRTLLM uniform FP8 KV memory budgeting (#41090) 2026-09-24 13:25:00 -07:00
36f59982fa MoE: small-batch sorting path with fused mxfp8 quantisation (#36559)
Co-authored-by: mikevin920 <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-09-24 13:18:18 -07:00
0b53305f48 MiniMax-M3: MXFP8 dense-only block convert + aiter MXFP8 MoE on gfx950 (#36574)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mikevin920 <mikevin920@yahoo.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-24 13:16:59 -07:00
Qiaolin Yu b129504f0e [qwen 3.8 next] Fuse Qwen PLE gate and convolution preparation for target verify (#40041) 2026-09-24 13:13:34 -07:00
Thomas Wang 1446e24d13 [AMD] Add .co for deepseek v4 fp8 decode kernel and add group decode opt (#41120) 2026-09-24 12:21:34 -07:00
Cheng Wan 5bb850d536 [Refactor] Split prepare_attn into a reduction step and per-quant-format residual steps (#41084) 2026-09-24 12:16:54 -07:00
Cheng Wan a80055d3c6 [Fix] Broadcast requests along attention CP before attention TP (#41083) 2026-09-24 12:16:27 -07:00
Cheng Wan 7c9f74c6e1 [Fix] Step-3.5: stop dense layers from summing their output twice under DP attention (#41082) 2026-09-24 12:15:58 -07:00
Cheng Wan 62ae032c46 [Refactor] Share the MoE output all-reduce between models (#41097) 2026-09-24 12:15:33 -07:00
Cheng Wan 5df1667077 [Refactor] Pass each layer stack's output through a communicator exit (#41081) 2026-09-24 12:15:07 -07:00
Cheng WanandMick Qian 07a3735e94 [Fix] Complete the deferred FFN all-reduce before deepstack addition and aux hidden-state capture (#41080)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-24 12:14:39 -07:00
Cheng Wan 928e683f94 [Fix] Complete the deferred FFN all-reduce before a pipeline-parallel send (#41079) 2026-09-24 12:13:57 -07:00
ChangLiu0709 6a14b80141 [AMD] GLM-5.2 MI355X MXFP4: bump image to 20260923 daily (#41109) 2026-09-24 12:02:01 -07:00
Khoa Pham 1b03d31b31 [Test] Run the Qwen3.5 Triton DCP nightly with the radix cache enabled (#41155) 2026-09-24 12:00:09 -07:00
f404db94d4 [DP attention] Publish DP buffer sizes from a ForwardBatch (#40858)
Co-authored-by: hanminglu <hanminglu@fb.com>
Co-authored-by: oss-sync bot (pranjalssh) <pranjalssh@users.noreply.github.com>
2026-09-24 11:34:19 -07:00
Mohammad Miadh AngkadandMohammad Angkad 86b3558b41 [Fix] Skip the DCP target-verify MLA kernel during FlashInfer autotune (#41138)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-24 11:19:28 -07:00
78b382b1ad Support unified memory decode host pools (#39478)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-09-24 11:02:15 -07:00
Trevor Morris 984994e3aa fix: Triton 3.8 compatbility to support DSV4.1-Flash in CUDA 13.4 image (Rubin) (#40805) 2026-09-24 10:25:35 -07:00
Jackey HuaandClaude Fable 5.1 5bb24e399d [Quant] ModelOpt mixed precision: dispatch block-FP8 MoE experts and derive the block size (#38726)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-24 12:30:01 -04:00
Xinyu JiangandZhiyao Jiang 8ec65e89c1 [AMD] ci: move the Miles ROCm 7.2 nightly build to 7.2.4 (#40387)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
2026-09-25 00:09:19 +08:00
Shangming CaiandClaude Fable 5 77173ffb08 [HiCache] Demote SWA KV to host on write_back eviction instead of dropping it (#40712)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-24 23:56:17 +08:00
Ting SUN 752801e4d1 fix(sampling): validate sampling_seed is an int within int64 range (#28960)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-09-24 17:25:13 +02:00
17a7484c06 [chore] surface the cookbook to users who pip install sglang (#40866)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-24 22:56:41 +08:00
Shangming CaiandClaude Opus 5 cd11037009 [PD] Enable deferred decode-side KV release by default (#41023)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 22:35:11 +08:00
黄孝君 4142235c2b [NPU] Update CANN version to 9.1.0 (#40524) 2026-09-24 21:35:31 +08:00
WenhaoZhang ec7eb6bd79 [diffusion] update code owner (#41130) 2026-09-24 21:31:50 +08:00
Xiaoyu Zhang ea5baf4022 [Refactor] Retire the model-specific Kimi K3 kernel namespace (#40922) 2026-09-24 21:25:00 +08:00
paulzhang-tm 28ec6700da [HiCache] Make host reclamation independent of transfer order (#40512) 2026-09-24 21:03:25 +08:00
3177d10ca6 Refactor the Cute-DSL AR fusion to support DeepseekV2 archs (GLM-5.3, etc.) (#39816)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
2026-09-24 06:00:58 -07:00
ronnie_zheng ce06a14444 [Diffusion] migrate the whole _register_configs from registry.py to the model own config file (#40612) 2026-09-24 19:29:49 +08:00
Xiaoyu Zhang c6565c1a6d [Diffusion] Enable lossless SANA-Video eager conv fusions for 12.6% lower latency (#40388) 2026-09-24 19:21:32 +08:00
Alan Kao a69583f70f [AMD] Add tuned dsv4 shape (#40996) 2026-09-24 04:06:40 -07:00
AMD-yanfeiwang 0a59830114 [AMD][DSV4] fp8 unified_kv decode: wave-aware split count past 40 tokens (#40878)
Signed-off-by: Yanfei Wang <yanfwang@crsuse2-m2m-v2-017.us-east2-a.compute.internal>
2026-09-24 03:41:23 -07:00
Xiaoyu Zhang 26c63335e2 [Diffusion] Fuse lossless LingBot World FP32 normalization (#40425) 2026-09-24 18:41:10 +08:00
Zhang, Jiejing d94d784441 [AMD] Tune Triton sparse MLA on gfx950 and make split-K workspaces graph-safe (#39059) 2026-09-24 03:21:53 -07:00
Mohammad Miadh Angkad bb1c98baa2 Fix TBO child batch missing dp_spec_prefill_coordination_applied (#41096) 2026-09-24 02:44:55 -07:00
Xiaoyu Zhang c85df5becc [Diffusion] Fuse lossless Wan VAE post-ops for LongLive 2 I2V (#40405) 2026-09-24 17:40:07 +08:00
Xiaoyu Zhang 261cb826e6 [Diffusion] Fuse LongCat GELU+cat and support Edit-Turbo BCG (#40384) 2026-09-24 17:27:56 +08:00
MickandMick Qian 174a5f37b9 [feature] add per-item candidate token scoring and calibration (#40826)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-24 17:00:13 +08:00
Cheng Xin 41812afe43 [NPU] Fix DSV4 hard-coding kv dtype (#40510) 2026-09-24 16:53:44 +08:00
Yihao Wang 81f2b43ab5 [ci] pr-gate: add generic require-label input and support pull_request_target (#40999) 2026-09-24 01:04:29 -07:00
pllimax d65503fece [NPU] Remove the LLaDA2.0-mini basic-function test case (#40905) 2026-09-24 15:58:48 +08:00
Yuwei An 1417345f5f [Experimental] Preserve speculative decoding during prefill across DP ranks (#40118) 2026-09-24 00:01:12 -07:00
Liangsheng Yin cae4927fbf [DSV4] Budget the ratio-2 pair state pool in DSV4PoolConfigurator (#41048) 2026-09-23 23:53:46 -07:00
Michael a164c6d9f1 [AMD] Register mem-cache unit tests in PR CI (#40812) 2026-09-23 23:35:51 -07:00
YC Yen-Ching TsengandBingxu Chen 43af9fcbb6 [AMD][DI][CI] Move MI355X disagg nightly to ROCm 10 (#41053)
Co-authored-by: Bingxu Chen <Bingxu.Chen@amd.com>
2026-09-24 14:24:39 +08:00
YC Yen-Ching Tseng 9f6fc55332 [AMD][Diffusion] FlyDSL fused norm kernels on wave32 targets (gfx1250) (#37751) 2026-09-23 23:06:55 -07:00
32290dda2c [AMD] Small-M MXFP4 fused-MoE kernel for gfx950 (Qwen) (#40204)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-23 22:45:33 -07:00
minke.yu 4f0ca95477 deploy: b300 build scripts with pip cache mount + no-build-isolation fast path
Base image already has torch 2.13, but pyproject build-system.requires
pulls it into pip's isolated build env on every build (~6GB via throttled
mirror, 47min). --no-build-isolation reuses the base env; build.sh picks
Dockerfile.fast (--no-deps) when python/pyproject.toml is unchanged.
2026-09-24 13:11:41 +08:00
Shuwen Wang 90663ccb41 [DSV4] fix: size the C4 state ring by the page it is addressed by (#40337) 2026-09-23 22:05:32 -07:00
cctryandcctry efff836911 [PD] Add decode host receive for custom transfer backends (#40238)
Co-authored-by: cctry <cctry@fb.com>
2026-09-23 21:57:33 -07:00
Theresa Shan 7a9feacf91 [Docs] DeepSeek-V4 MI355X Pro Official PD pairs with DSpark and UMBP (#41024) 2026-09-23 21:48:27 -07:00
fanxingranandThomas Wang 0dc8b29e15 [ROCm][DSA] Enable AITER fused FP8 indexer writer (#40710)
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-09-23 21:46:35 -07:00
Nan Jiang c3685dff42 [Sampling] Add selected/support sampling logprob modes (#40932) 2026-09-23 21:38:19 -07:00
Zhaoyi Li 8eedf616f5 [AMD][DI][CI] Say which image the MI355X nightly ran on (#41026) 2026-09-24 12:18:03 +08:00
minke.yu b91137ab98 scheduler: CP-symmetric idle check for health-check admission
build-sglang-image / build (push) Successful in 27m52s
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.

Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).
2026-09-24 12:01:57 +08:00
minke.yu 67d8368a84 deploy: archive b300 ds41 compose files; skip CI for deploy/ and .gitea/ changes 2026-09-24 11:49:37 +08:00
Jimmy Shong be495a6174 [Fix] Use cached prefix lengths for FlashInfer full-attention ragged prefill (#40796) 2026-09-23 17:23:25 -10:00
jacky.cheng e2f4fedf04 [AMD] Critical fix enabling Qwen3.8 FP8: restore dropped fused shared-expert weights (#40754) 2026-09-23 20:21:33 -07:00
Shijin ZhangandBrayden Zhong 77983865d8 [Moe] Honor swiglu_limit clamped activation in flashinfer_cutlass runner (#39939)
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2026-09-23 20:10:03 -07:00
Xia WeiwenandCopilot 5c44214d6a [XPU] Qwen3.8-flash-next enablement (#37213)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-24 11:06:30 +08:00
f4d9d2e670 [RL] Add RL weight-update sessions and support updating spec draft runners (#40777)
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Nan Jiang <59716405+nanjiangwill@users.noreply.github.com>
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: JD-ETH <jaedon.guo@gmail.com>
Co-authored-by: Letian Ruan <ruanlt420@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-23 20:04:57 -07:00
Yueming Yuanandzyzshishui f4b90382a2 [RL] Keep DSA cuda-graph state and the graph pool intact across TMS pause/resume (#40804)
Co-authored-by: zyzshishui <zyzshishui@gmail.com>
2026-09-23 19:55:26 -07:00
Liangsheng Yin cc6e8b0f25 [Fix] Derive per-runner hybrid SWA layer ids on ModelLayerInfo instead of mutating ModelConfig (#40983) 2026-09-23 19:52:55 -07:00
jacky.cheng e98b2f7538 [AMD] Speed up Wan2.2 DiT FP8 attention per-tensor quantization (#34695) 2026-09-23 19:49:26 -07:00
jacky.cheng 8b5d77c268 [AMD] Add a Triton packed sparse decode path for QSA on ROCm (#38876) 2026-09-23 18:59:17 -07:00
amote-i 82cdd72588 [NPU] [DOC] Remove --enforce-shared-experts-fusion from npu docs (#41000) 2026-09-24 09:54:29 +08:00
hanwlax 03fcbe147f [NPU] Support batch invariant FIA graphs for deterministic inference (#39607) 2026-09-24 09:53:17 +08:00
Wang, FangYuan 81b81664a6 Revert "[AMD] Fix DeepSeek-V4 accuracy by not passing num_token_non_padded to MoE topk" (#40946) 2026-09-23 18:38:19 -07:00
Liangsheng Yin 621136e094 [mem_cache] Remove unused helpers in mem_cache, storage backends, and metrics (#40963) 2026-09-23 18:16:44 -07:00
Liangsheng Yin 81cb895b85 [Fix] Recover from stale torch extension locks in every cpp_extension loader (#40989) 2026-09-23 17:53:57 -07:00
ec75d3d30f MiniMax-M3: allocate the lightning-indexer K cache in fp8 on gfx95 (#36549)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-23 17:52:02 -07:00
zijiexiaandClaude Opus 5.5 ffac53d779 [Doc] Add H200 recipes to MiMo-V2.6 cookbook (#40969)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 16:49:35 -07:00
Yueming Yuan 542a043f1b [RL] Release the weight-checker snapshot once compare passes (#37284) 2026-09-23 16:06:02 -07:00
James LiuandClaude Opus 5.5 208f6f7501 [Spec] Support DFLASH for Kimi K3 (#40794)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 15:58:39 -07:00
Alison Shao 3f697899a3 ci: reinstall torch/triton left incomplete by a cancelled job (#40819) 2026-09-23 15:55:02 -07:00
Cheng Wan 0010f56a0d [Test] Remove obsolete configuration migration guards (#40976) 2026-09-23 15:33:54 -07:00
6fe4b66f5c [RL] Keep pause_generation and weight updates from deadlocking each other (#40779)
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Zhichenzzz <zczeng@uw.edu>
2026-09-23 15:13:42 -07:00
Cheng Wan 79fec592a8 [Refactor] Read parallel placement in consumers (#40638) 2026-09-23 14:57:25 -07:00
f2a1366584 MiniMax-M3: wave64 histogram-select decode top-k, and raise kMaxNumBlocks for CUDA graphs (#36560)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
2026-09-23 14:50:13 -07:00
4fa2c9c1de MiniMax-M3: run the sparse prefill main attention through AITER Gluon paged attention (#36546)
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Alex Sun <alex.s@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@gmail.com>
Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
2026-09-23 14:49:10 -07:00
Cheng Wan f766397f2e [Refactor] Trim server configuration and runtime context comments (#40971) 2026-09-23 14:47:13 -07:00
Liangsheng Yin c39a1c0663 [mem_cache] Free the rows below the SWA evict floor on all-SWA request release (#40798) 2026-09-23 14:46:28 -07:00
Cheng Wan ab0c31bd94 [Refactor] Move eleven more MoE models to LayerCommunicator.ffn_exit (#40871) 2026-09-23 14:43:41 -07:00
Cheng Wan e1048a215c [Refactor] Let LayerCommunicator own the FFN exit in Qwen3-MoE, DeepSeek-V2 and GLM4-MoE (#40870) 2026-09-23 14:42:58 -07:00
Cheng Wan 7dd9640752 [Refactor] Compare token layouts instead of group sizes when selecting communicator paths (#40869) 2026-09-23 14:41:55 -07:00
Cheng Wan e31cbc01a2 [Fix] Stop deferring the last layer's FFN all-reduce in five models (#40868) 2026-09-23 14:41:14 -07:00
Cheng Wan 701cf7e47e [Refactor] Run Nemotron-H DP attention through the standard layer communicator (#40867) 2026-09-23 14:40:33 -07:00
Cheng Wan 379e8f916b [Fix] Capture complete Nemotron auxiliary hidden states (#40801) 2026-09-23 14:39:50 -07:00
Cheng Wan 9ebe42238b [Fix] Reduce Nemotron MTP attention outputs once (#40800) 2026-09-23 14:39:04 -07:00
Cheng Wan 0e80c73c92 [Fix] Avoid duplicate residual in LongCat MoE shortcut (#40799) 2026-09-23 14:37:39 -07:00
Liangsheng Yin 954458567e [mem_cache] Remove unreachable RadixCache paths in KV canary and HiCache accessors (#40807) 2026-09-23 13:22:58 -07:00
2909f84e3b [JIT] Add an occupancy-preserving L1 carveout preference (#40767)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Oguz Ulgen <oulgen7@gmail.com>
2026-09-23 11:49:34 -07:00
890a9605ce [Metrics] Log forward and forward+idle occupancy over total wall time (#40802)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-23 11:48:31 -07:00
4bb5611dba [Fix] Give the full prefill CUDA graph replay view the captured bucket's input_ids (#40851)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: fei-xx <135589532+fei-xx@users.noreply.github.com>
2026-09-23 11:48:03 -07:00
f2f223e697 Allow attention layers to opt out of the prefill wrapper (#40683)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
2026-09-23 11:39:22 -07:00
Byron Hsu f2eebd5533 [RL] Fix Kimi K3 expert-count lookup for routed-expert capture (#40700) 2026-09-23 11:18:04 -07:00
7fef014d51 feat(kv-hints): add kv hint envelope to request transport (#38891)
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
2026-09-23 10:31:13 -07:00
Even Zhou 5c154c214d Revert " [NPU] Enable piecewise CUDA graph support on NPU" (#40895) 2026-09-23 23:38:14 +08:00
Xiaoyu Zhang c3e985203a [Diffusion] Remove unused standalone benchmarks and deduplicate kernel tests (#40924) 2026-09-23 22:42:03 +08:00
Xiaoyu Zhang 8993f790be [Diffusion] Fuse lossless SenseNova RoPE for 5% faster H200 inference (#40374) 2026-09-23 22:37:22 +08:00
Xiaoyu Zhang 4e60d70ba4 [Diffusion] Fuse Joy Image Edit QKV concatenation and avoid QK copies (#40494) 2026-09-23 22:20:14 +08:00
Xiaoyu Zhang 172b1b4825 [Diffusion] Accelerate Cosmos3 Edge on Hopper with lossless fusions (#40386) 2026-09-23 21:16:58 +08:00
iridiumine 16e353d93e [NPU] Skip fused gmm1+swiglu for swiglu_limit (SiLU-with-clamp) checkpoints (#40438) 2026-09-23 21:02:54 +08:00
iridiumine 3fdd63a562 [NPU] Fuse FIA KV-cache K/V writes into one npu_scatter_pa_kv_cache call (#40445) 2026-09-23 21:02:39 +08:00
Xiaoyu Zhang abef3efb64 [Diffusion] Fuse rounded SwiGLU for quantized MiniMax-H3 MLPs (#40378) 2026-09-23 19:55:48 +08:00
minke.yu db7d2cb7db fix: vision check uses get_parallel().pp_group (get_pp_group undefined in this tree)
build-sglang-image / build (push) Successful in 27m58s
2026-09-23 18:05:57 +08:00
injet f23179ce99 更新 .gitea/workflows/build-image.yaml
build-sglang-image / build (push) Successful in 26m22s
2026-09-23 17:54:53 +08:00
injet 8266769b2d 更新 .gitea/workflows/build-image.yaml
build-sglang-image / build (push) Failing after 1m12s
2026-09-23 17:49:30 +08:00
injet 3b671d6086 更新 .gitea/workflows/build-image.yaml 2026-09-23 17:48:46 +08:00
hanwlax 2d25767759 [NPU] Enable piecewise CUDA graph support on NPU (#28417) 2026-09-23 17:30:30 +08:00
inkcherryandDuyi-Wang 58f622ebc4 Reduce decode bootstrap latency with request-owned speculative KV (#38978)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:24:53 -07:00
TianDi101andDuyi-Wang 4cd63da996 [Unified Cache] Dedup replicated MLA/DSA KV in the UMBP direct linker (#38778)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:15:01 -07:00
a89f849158 [ROCm] Fuse the MLA q absorb into the RoPE + KV-write kernel on gfx950 (#38340)
Co-authored-by: JohnQinAMD <zhengsheng1997@gmail.com>
Co-authored-by: RuibinCheung <ruibzhan@amd.com>
2026-09-23 02:07:54 -07:00
siliangchen-amd 401d5aedf3 [AMD] Drop the unreachable vLLM fallback from ROCm FP8 activation quant (#40879) 2026-09-23 01:59:15 -07:00
Yoray Zack de123f38bb [3/N] elastic-ep: Recapture decode CUDA graphs after scale-up (#33723) 2026-09-23 01:51:30 -07:00
minke.yu 7925735a3e ci: also trigger image build on dsv41-pd-visioncp pushes
build-sglang-image / build (push) Successful in 29m50s
2026-09-23 16:46:06 +08:00
aa0b65c7c7 [AMD] Fix DeepSeek-V4 accuracy by not passing num_token_non_padded to MoE topk (#39804)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
2026-09-23 01:35:56 -07:00
RuibinCheungandxiaobochen-amd 48c3854620 [ROCm] feat: enable aiter allreduce fusion for GLM models (#39790)
Co-authored-by: xiaobochen-amd <xiaobo.chen@amd.com>
2026-09-23 01:24:08 -07:00
MatsueYu d4dcce12d4 [npu] decoding procedure optimization on qwen3.5/3.6 (#35958) 2026-09-23 15:23:51 +08:00
ZiruandNiu Ziru 0b0f947265 [CPU] Add fused_sigmod_mul_cpu operators to the Meta Muse Glimmer model. (#39538)
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
2026-09-23 15:12:43 +08:00
Xinyuan Tong 6833498646 model: prune comments and redundant tests in dsv41 vision CP 2026-09-23 14:36:01 +08:00
Xinyuan Tong b48e2cb1eb model: TP-wide single-owner image encoding for DeepSeek V4.1
ViT and Aligner are replicated per TP rank, encoding each image eight
times with TP8 on both CP1 and CP8. Elect one owner per image and use
ordered full-span broadcasts with a six-phase agreement protocol.
Both CP1 and CP8 benefit while local cache hits preserve collective order.
2026-09-23 14:36:01 +08:00
Xinyuan Tong bfeb7cd9b2 model: support DeepSeek V4.1 vision with interleave prefill CP
The CP runner bypassed the vision merge and used bare text embeddings.
Merge image features before sharding so request-global offsets stay valid.
Canonicalize model IDs separately to preserve scheduler hash IDs.
Keep unsupported combinations guarded and isolate embedding overrides
from multimodal prefills without starving queued FCFS requests.
2026-09-23 14:36:00 +08:00
Mohammad Miadh AngkadandMohammad Angkad 86cb4a0f97 [CI] Update GLM-5.3-Flash H200/B200 test args (#40862)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-22 23:21:04 -07:00
Polisetty V R K Jyothendra Varma abca3b2e52 [Intel GPU] Add DeepSeek-V2-Lite-Chat-FP8 gsm8k e2e accuracy nightly test on XPU (#39781)
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com>
2026-09-23 14:11:30 +08:00
ANSHUMAN TRIPATHY eb5e8c84be Speculative Decoding with NGRAM support for XPU (#31362) 2026-09-23 14:04:20 +08:00
Zhaoyi Li aa0feca536 [AMD][DI][CI] Use a node-local model cache on the SPUR cluster (#40813) 2026-09-22 23:01:55 -07:00
Shuwen Wang 66ce8c55cc [HiCache] ci: add HiCache and unified radix rerun group (#40831) 2026-09-23 13:56:24 +08:00
a3d11f9d95 [diffusion] feat: support permanent lifetime for layerwise resident layers (#40599)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 13:49:09 +08:00
28be39f72e [deepep_v2] support GLM-5.3-Flash (Glm5NextForConditionalGeneration) (#40466)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 22:32:34 -07:00
jacky.chengandEddie A. Tsai 58988bee2f [AMD] Add diffusion (Wan2.2) extras to gfx1151 Docker image (#40844)
Co-authored-by: Eddie A. Tsai <56776714+vvvdwbvvv@users.noreply.github.com>
2026-09-22 22:18:55 -07:00
Jeff HansonandJeff Hanson 5b8d8b2f88 [Fix] Keep Inkling automatic tool grammar active across the response (#40468)
Co-authored-by: Jeff Hanson <2469458+jshanson7@users.noreply.github.com>
2026-09-23 13:14:36 +08:00
Shuwen Wang ced0a2c98c [Hisparse] fix: account for MiniMax HiSparse full-pool memory (#40743) 2026-09-23 13:13:48 +08:00
Eric Zhang 9d267655e3 Fix NIXL transfer of MXFP8 KV block scales (#40792) 2026-09-23 13:12:04 +08:00
973fb44471 [Diffusion] Support MiniMax-H3 PDD(Parallel Decoding Distillation) inference (#40568)
Co-authored-by: Forrest Lin <forrestl@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-23 12:57:26 +08:00
Shuwen Wang 1d59ce7c90 fix: partition selective CI reruns into matrix jobs (#40760) 2026-09-22 20:37:02 -07:00
zhaozx-cn 525f14040d [NPU]fix ci hicache oom (#40824) 2026-09-23 11:16:50 +08:00
Shuwen Wang 224a247d5b [HiSparse] ci: add cross-directory rerun test group (#40749) 2026-09-23 11:07:03 +08:00
c2bc456b64 [chore] point agents at the cookbook before test configs (#40818)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-23 10:54:15 +08:00
Shuwen Wang 2bc435d32e [HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler (#40312) 2026-09-23 10:52:03 +08:00
Kan WuandClaude Fable 5.1 2e3568567b [sgl-router] Launch reorg routing with existing policy options (#40766)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 10:18:11 +08:00
Richard Wang 40048f6e51 [dLLM] feat: support DiffusionGemma serving (#34061) 2026-09-23 09:59:31 +08:00
Liangsheng Yin 78980a3b0b [misc] Remove deprecated endpoints, env vars and aliases past two releases (#40795) 2026-09-22 18:14:45 -07:00
Alison Shao 0cd8be351d ci: stop Runner Utilization Report from draining the shared API quota (#40791) 2026-09-22 18:01:40 -07:00
Qi SunandXiaoyu Zhang 06008c170d [dsv4.1]Optimize FP4 indexer by skipping invisible tiles (#40431)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-23 07:48:43 +08:00
Liangsheng Yin 4ce23542bf [HiCache] Remove the unused HiRadixCache (#40787) 2026-09-22 16:17:12 -07:00
Mohammad Miadh AngkadandMohammad Angkad 7b977ce5dc [Fix] Decide the MoE padded-row bound from the layer scatter mode (#40672)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-22 16:02:17 -07:00
Liangsheng Yin d34f7b2326 [mem_cache] Clean up SWA/Mamba radix cache leftovers and drop SGLANG_ENABLE_UNIFIED_RADIX_TREE (#40780) 2026-09-22 15:40:57 -07:00
19ee4b566b fix(function_call): buffer complete DeepSeek DSML invokes (#39632)
Co-authored-by: Xinyuan Tong <justintong0323@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-22 15:38:44 -07:00
Jialin Ouyang b77833c504 [MM] Keep scheduler padding in packed token arrays (#40357) 2026-09-22 15:12:07 -07:00
c19dc43cc7 [AMD] [GLM-5.3-Flash Day 0] Load the MXFP4 MTP draft layer (#39779)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: CHANG <jacchang@crsuse2-slog-003.us-east2-a.compute.internal>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-22 15:07:51 -07:00
jain-ria 01275aab6b fix(grpc): expose native response timeout as a server argument (#40644)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-22 14:57:38 -07:00
Liangsheng Yin 9a53f75124 [mem_cache] Remove the experimental C++ radix tree (#40775) 2026-09-22 14:19:45 -07:00
Faradawn Yang d72629e4ba Add GB200/GB300 hardware to Qwen3.5 (#40770)
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
2026-09-22 14:16:43 -07:00
sglang-botandsglang-bot c08ffef709 docs: sync LMSYS SGLang blog cards (#40655)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-09-22 21:14:07 +00:00
Xinyuan TongandZijie Xia 542c817bfa [Docs] Fix benchmark table column overflow in cookbook deployment panel (#40114)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
2026-09-22 14:07:47 -07:00
3afdde5f05 [AMD] [GLM-5.3-Flash Day 0] Enable the k-pool DSA indexer on gfx950 (#39341)
Co-authored-by: Yikai Zhang <ykzhang12@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
2026-09-22 13:14:30 -07:00
077c319914 [AMD] [GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm (#39778)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-22 13:10:39 -07:00
d6cc283da7 [AMD] [GLM-5.3-Flash Day 0] Enable zero-RoPE TileLang DSA on gfx950 (#38547)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-22 13:08:25 -07:00
Yuwei An 4cbf290fb9 [Fix] Handle chunked paged MQA metadata in DSV4.1 eager forwards (#40637) 2026-09-22 12:59:04 -07:00
720617bb5b [AMD] Reuse KV gather indices across ASM context prefill layers (#39901)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 12:08:05 -07:00
5c18780c31 [Router] Keep e2e workers inside the job's CUDA_VISIBLE_DEVICES allotment (#40686)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 11:32:12 -07:00
Cheng Wan db73f35f4a Fix the Inkling per-expert sync test and collect it in the weekly CPU run (#40725) 2026-09-22 11:22:07 -07:00
Shangming Cai 6f4c2b9b91 [HiCache] Demote internal-node mamba states on write_back eviction (#40680) 2026-09-23 02:18:58 +08:00
Cheng Wan 91c329cc86 Take the model config out of the parallel group build, and finish retiring the parallel getters (#40707) 2026-09-22 10:55:21 -07:00
Pei Li 66454d704b [Feature] Support --tokenizer-worker-num > 1 in the offline Engine API (#40260) 2026-09-22 10:50:01 -07:00
Sage b4701d672d [rust-renderer] decouple renderer sampling from protocols (#40747)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 10:48:28 -07:00
Sage 8ab21c8a94 [ci] publish renderer image (#40639)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 09:48:12 -07:00
Sage a7b96fdad7 [ci] run cpu ci for renderer-only changes (#40636)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-09-22 09:47:48 -07:00
YAMYandzhangxiaolei 6fd98c98b9 [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp (#40501)
Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
2026-09-22 08:46:44 -07:00
c79510cc2a [DSv4.1] Score prefill consumer index layers on candidate blocks with DeepGEMM (#40352)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com>
2026-09-22 22:06:06 +08:00
4a1b69abc8 [diffusion] docs: correct the resident-layer help text to match its scope (#40593)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 20:32:59 +08:00
pllimax 244db08d60 [NPU][CI] Constrain evalscope dependency versions (#40133) 2026-09-22 20:15:11 +08:00
Wookje Han bc3a63ebdb [Fix] Missing SWA eviction during decode preallocation (#40309) 2026-09-22 20:11:40 +08:00
Shunkangz ddebc52f23 [kv-shard 3/4] Enable Control plane (#38468) 2026-09-22 19:57:44 +08:00
2524b61f9b [diffusion] feat: allow a component use retain its layerwise resident set (#40592)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 19:56:21 +08:00
Shangming Cai 861b11f087 [PD] Simplify late-abort quiescent ack branch to else (#40711) 2026-09-22 19:08:47 +08:00
amote-i cb8dab06be [NPU] [DOC] Remove duplicated features in npu docs (#40714) 2026-09-22 19:04:17 +08:00
cctry 8ef6d31cb9 [PD] Preserve abort ACKs until in-flight KV transfers drain (#40645) 2026-09-22 18:46:07 +08:00
Emre AlbayrakandXiaoyu Zhang 9d58189c12 [diffusion] attention: add fp8_fa_sm120 FP8 backend for SM120 GPUs (#40175)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-22 17:44:37 +08:00
debbb5cde9 [AMD] Drop the redundant scale zero-fill before AITER per-tensor FP8 quant (#40557)
Co-authored-by: siliangchen-amd <{"message":"Although you appear to have the correct authorization credentials, the `amd-eng-emu` enterprise has an IP allow list enabled, and your IP address is not permitted to access this resource.","documentation_url":"https://docs.github.com/rest/users/users#get-the-authenticated-user","status":"403"}siliangchen-amd@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-09-22 17:43:31 +08:00
Michael 3f00fb7e2e [AMD] Register unified KV page-zeroing test in PR CI (#40123) 2026-09-22 02:16:31 -07:00
Zhaoyi Li 70a2bb2a30 [AMD][DI] Keep loopback in UCX_NET_DEVICES (#40641) 2026-09-22 02:15:18 -07:00
jacky.chengandchuyeh 790551c382 [AMD] Skip full-vocab softmax in EAGLE topk==1 draft on ROCm (#35872)
Co-authored-by: chuyeh <chuyeh@users.noreply.github.com>
2026-09-22 01:57:37 -07:00
chuyehandjacky.cheng c2f14bfbc4 [AMD] Use exact CU share for gfx950 segment-plan headroom (#39503)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 01:49:59 -07:00
inkcherry 367e3700cf avoid host sync in DSpark prefill slot expansion (#40111) 2026-09-22 01:20:49 -07:00
d00adfd19c [AMD] Fix deferred Kimi-K3 forget gate in fused in-projection (#39525)
Co-authored-by: Tom Jen <tomjen12@localhost.localdomain>
Co-authored-by: HAI <hixiao@gmail.com>
2026-09-22 01:07:25 -07:00
6412ad8c64 [AMD] Pack Qwen3.5 GDN input projections on ROCm (#39902)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-09-22 00:52:40 -07:00
Po-Han Huang (NVIDIA) 771c9d782d [PD] Validate Mooncake EFA allocator compatibility (#39973) 2026-09-22 15:12:29 +08:00
Suraj 1815e490de [observability] Fix negative queue_time for retracted requests (#39312) 2026-09-22 15:08:01 +08:00
jshn9515 c948114d58 [DOC] Update quickstart guide to use sglang serve for launching the server (#40647) 2026-09-22 15:00:35 +08:00
5488 changed files with 671247 additions and 143346 deletions
@@ -1,10 +1,14 @@
---
name: add-jit-kernel
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang.kernels JIT infrastructure and public operator groups
---
# Tutorial: Adding a New JIT Kernel to SGLang
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -497,6 +501,32 @@ def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) ->
---
## Register the public entry point (required)
Add metadata in `python/sglang/kernels/ops/elementwise/__init__.py` so the
operator appears in the registry without importing its implementation:
```python
from sglang.kernels.registry import register_kernel
from sglang.kernels.spec import CapabilityRequirement, KernelBackend, KernelSpec
register_kernel(
KernelSpec(
op="elementwise.scale",
backend=KernelBackend.JIT,
target="sglang.kernels.ops.elementwise.scale:scale",
capabilities=frozenset({CapabilityRequirement.CUDA}),
)
)
```
Record the devices the implementation actually supports; the CUDA-only scale
example above does not imply that every JIT kernel is CUDA-only. Runtime and
correctness tests import `scale` from `sglang.kernels.ops.elementwise.scale`.
Do not import the implementation eagerly in the group initializer. When an
existing `BaseFusedOp` owns the operation, add the implementation there instead
of registering a second conflicting op/backend pair.
## Step 3 (optional): Tune JIT build flags
If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
@@ -611,6 +641,8 @@ if __name__ == "__main__":
Benchmarks are `bench_*.py` files under `test/registered/kernels/benchmark/<group>/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/jit/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
@@ -619,7 +651,7 @@ Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/j
- `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
- `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
- Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
- `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"`; if you narrow it, keep every read, in-place, and output tensor.
- `use_cuda_graph=False` for kernels that can't be captured.
- `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
- `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
@@ -724,7 +756,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
- `python/sglang/kernels/jit/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
- `python/sglang/kernels/jit/include/sgl_kernel/atomic.cuh` — `atomic::max`
- `python/sglang/kernels/jit/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
- `python/sglang/kernels/jit/csrc/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh` — minimal runnable reference
- `python/sglang/kernels/jit/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
- `python/sglang/kernels/jit/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
- `python/sglang/kernels/jit/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
@@ -5,6 +5,10 @@ description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel
# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
Apply [kernel-organization](../kernel-organization/SKILL.md) for the public
operator namespace, logical grouping, lazy registry metadata, and test placement.
The implementation tutorial below does not replace that API contract.
This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
## Goal
@@ -25,7 +29,7 @@ Add a new operation that scales each element of a tensor by a scalar factor:
In addition, every new kernel must ship with:
- **Tests** (pytest)
- **A benchmark script** (triton.testing)
- **A benchmark script** (`marker.do_bench` from `sglang.kernels.jit.benchmark` — it works for any callable, not only JIT kernels)
---
@@ -194,6 +198,21 @@ Then re-export it from `python/sglang/kernels/aot/python/sgl_kernel/__init__.py`
---
## SGLang integration (required for runtime use)
After exposing the AOT wheel symbol, add a lazy wrapper and a `KernelSpec` under
`python/sglang/kernels/ops/<group>/`. Set `backend=KernelBackend.AOT`, record
actual device/architecture support, and keep `sgl_kernel` imports inside the
implementation path. SGLang runtime and integration tests import that wrapper.
For this example, use `sglang.kernels.ops.elementwise.scale`.
The wheel-level tests below validate its standalone API/build. They do not
replace CI-registered SGLang correctness tests in
`test/registered/kernels/ops/elementwise/test_scale.py` and benchmarks in
`test/registered/kernels/benchmark/elementwise/bench_scale.py`. Follow
`write-sglang-test` for their registration and CI budget; do not register tests
under the shipped `python/sglang/` package.
## Step 6: Write tests (required)
Create `python/sglang/kernels/aot/tests/test_scale.py`:
@@ -241,62 +260,46 @@ if __name__ == "__main__":
## Step 7: Add a benchmark (required)
Every benchmark must account for L2 cache reuse — see [`rules/kernel-benchmark.md`](../../rules/kernel-benchmark.md).
Create `python/sglang/kernels/aot/benchmark/bench_scale.py`:
```python
import itertools
import torch
import triton
import triton.testing
import sgl_kernel
from sglang.utils import is_in_ci
IS_CI = is_in_ci()
dtypes = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
sizes = [4096] if IS_CI else [2**n for n in range(10, 20)] # 1K … 512K
factors = [2.0]
configs = list(itertools.product(dtypes, sizes))
from sglang.kernels.jit.benchmark import marker
def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
return input * factor
def sglang_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
sgl_kernel.scale(input, factor, out=out)
@triton.testing.perf_report(
triton.testing.Benchmark(
x_names=["dtype", "size"],
x_vals=configs,
line_arg="provider",
line_vals=["sglang", "torch"],
line_names=["SGL Kernel", "PyTorch"],
styles=[("green", "-"), ("red", "--")],
ylabel="µs (median)",
plot_name="scale-performance",
args={},
def torch_scale(input: torch.Tensor, factor: float, out: torch.Tensor) -> None:
torch.mul(input, factor, out=out)
FN_MAP = {"sglang": sglang_scale, "torch": torch_scale}
@marker.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32], [torch.float16])
@marker.parametrize("size", [2**n for n in range(10, 20)], [4096]) # 1K .. 512K
@marker.benchmark("provider", ["sglang", "torch"])
def benchmark(dtype: torch.dtype, size: int, provider: str):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
return marker.do_bench(
FN_MAP[provider],
# Pass every tensor through input_args (not a closure) so marker rotates
# them across CUDA-graph calls to defeat L2 reuse.
input_args=(input, 2.0, out),
# Bandwidth = bytes(input) + bytes(out), both already in input_args.
memory_output=None,
)
)
def benchmark(dtype, size, provider):
input = torch.randn(size, dtype=dtype, device="cuda")
out = torch.empty_like(input)
factor = 2.0
if provider == "sglang":
fn = lambda: sgl_kernel.scale(input, factor, out=out)
else:
fn = lambda: torch_scale(input, factor)
ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
fn, quantiles=[0.5, 0.2, 0.8]
)
return 1000 * ms, 1000 * max_ms, 1000 * min_ms
if __name__ == "__main__":
benchmark.run(print_data=True)
benchmark.run()
```
---
@@ -167,7 +167,7 @@ describes what each one does.
3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast)
4. If all matched jobs are completed and count matches `expected_count` → success
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
6. Timeout after `max-wait-minutes` of wall-clock time (240 min for base-a, 350 min for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends `cancelled`, not `timeout`, and `timeout-minutes` cannot raise it. The base-b wait sets `fail-on-timeout: 'false'`, so a timeout lets base-c start; `pr-test-finish` fails the run on any base-b failure regardless
**Job specs example** (base-b):
```json
@@ -378,7 +378,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|---------|-------------|---------------|
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation, and whether a `wait-for-base-*` job hit the 6h hosted-runner cap |
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
@@ -396,7 +396,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` |
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
@@ -264,5 +264,5 @@ git add docs/src/snippets/configs/<hf-org>/<slug>.jsx \
docs/docs.json docs/cookbook/<category>/intro.mdx
git commit -m "Add <Display-Name> cookbook"
git push -u origin add-<model>-cookbook
gh pr create --title "Add <Display-Name> cookbook" --body "..."
gh pr create --title "[Doc] cookbook: Add <Display-Name> cookbook" --body "..."
```
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
`{{PORT}}` placeholders so users can override through the Env modal.
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
then parallelism (`--tp`, `--attn-dp-size`), then MoE
flags, then tuning knobs, with `--host` / `--port` last. The playground
engine assumes this ordering when inserting overrides (its anchors target
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
| Axis key | Widget | Use when |
|---|---|---|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
knobs/values that are disabled under the live facts, so stale picks never
emit a blocked combination. Interleave prefill-CP + DP-Attention is
deliberately NOT grayed (combined support is planned upstream even though
current releases assert `dp_size == 1` for interleave) — the engine shows
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
a warning hint below the command box instead.
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
playgroundFeatures: {
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
attention: {
knobs: [
{ id: "tp", label: "TP", values: [
@@ -357,8 +357,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -403,8 +402,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -13,8 +13,8 @@ import { config } from '/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx';
## 1. Quick start
Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose
a verified recipe below.
Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
then choose a verified recipe below. Docker is recommended for Linux GPU deployments.
<Deployment config={config} />
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
can't do coupled changes), and so does a toggle the legacy page itself labels
with operating-point words — e.g. a `dpattention` radio whose options are
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
even when its flags are uncoupled (`--attn-dp-size N` is a pure
flag add). Any other toggle that only adds/removes its own flags becomes a
Playground axis with the flags baked into cells when the legacy default was
ON — EXCEPT parsers:
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
Caveats discovered in the pilot:
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
@@ -136,8 +136,8 @@ than restating.
= `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
column must be produced by the SAME harness — flag a page that, say, measures one
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
platform's GSM8K with a historical raw completion harness and another's with
`sgl-eval run gsm8k --num-examples 1319` and shows both as one "GSM8K %"
(the scores aren't comparable). Either standardize on one harness (matching
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
@@ -162,7 +162,7 @@ Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only f
For a rename where the old key must keep working with a warning:
```python
SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME")
```
Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
@@ -11,6 +11,7 @@ This skill launches an SGLang server, validates it with a quick accuracy test, g
- A working SGLang installation (`pip install -e .` or equivalent)
- At least one available CUDA GPU
- The `sgl-eval` CLI, installed in its own virtualenv as described in `.claude/rules/prefer-sgl-eval.md`, not in the server's env
## Step-by-step Workflow
@@ -46,13 +47,11 @@ Typical startup time: 30-90 seconds depending on model size and whether CUDA gra
### Step 3: Validate accuracy (sanity check)
```bash
python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 20
```
- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- Expected `score`: **> 80%** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
- This is a quick sanity check, not a rigorous benchmark.
- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
### Step 4: Generate the profile
@@ -110,8 +109,8 @@ for i in $(seq 1 120); do
done
# 3. Accuracy check
python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
# Expected: Accuracy > 0.8
sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --num-examples 20
# Expected: score > 80%
# 4. Profile
python3 -m sglang.test.send_one --profile
@@ -0,0 +1,82 @@
---
name: kernel-organization
description: Apply the SGLang kernels RFC when adding, moving, splitting, or reviewing kernel APIs, registry metadata, kernel tests, benchmarks, and model-specific implementations. Use with add-jit-kernel, add-sgl-kernel, and write-sglang-test for placement and migration checks.
---
# Kernel organization
Read `python/sglang/kernels/README.md` and `test/README.md` before choosing a
location. [RFC #29630](https://github.com/sgl-project/sglang/issues/29630)
establishes the namespace; subsequent migrations #32148 and #40922 clarify
logical grouping and remove the old `_jit_` filename prefix.
## Choose the owner
- Put callable SGLang kernel APIs under `sglang.kernels.ops.<group>`. Runtime
and integration tests import from that namespace, including its submodules.
The `sgl_kernel` wheel retains its own public API and packaging tests.
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
routing in `moe`, attention index selection in `attention`, sampling in
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
quantization alone belongs in `quantization`.
- Model-specific files/subpackages and tuning data are allowed **inside** a
logical group. Do not add a model bundle or an implementation file directly
under `ops/`. Propose a new logical group only for a distinct responsibility,
and update `ops._GROUPS`, documentation, and tests together.
- Classify a fused operator by its complete contract. Do not split a fused
kernel into separate launches just to separate norm, RoPE, or quantization.
Split unrelated public entry points that happen to share a source file.
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
wrappers call it from their logical group. Keep process groups, communicator
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
`srt/layers/communication/` need not pretend to be generic interfaces.
## Register the public entry point
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
existing `BaseFusedOp` registration when the operation has interchangeable
backends. Group imports must not import GPU implementations or compile kernels.
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
existing backend dispatch and describe actual device/architecture restrictions
with `CapabilityRequirement`; JIT/AOT are not device types. Input shape/dtype
checks remain part of the entry point's contract. Registering a torch custom op
does not register it in the SGLang kernel inventory. Predicates, private JIT
factories, reference helpers, and runtime classes are not separate kernel APIs.
## Place and preserve tests
- Kernel numerical tests: `test/registered/kernels/ops/<group>/`.
- CI microbenchmarks: `test/registered/kernels/benchmark/<group>/`.
- Inventory/selector tests: `test/registered/unit/kernels/`.
- Runtime unit tests: `test/registered/unit/<subsystem>/`, following the tested
runtime module. A CUDA allocation does not make a runtime test a kernel test.
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
standalone test/benchmark entry points in production operator modules.
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
execution entry points, and CI stages/runners. Apportion existing time estimates
across split files; do not duplicate the original budget for every output file
or silently drop a registration. Follow `write-sglang-test` for CI registration.
## Verify a migration
1. Search all runtime, test, benchmark, documentation, patch-string, and lazy
registry references before deleting the old path. Check relative imports and
package re-exports as well as direct imports. Do not leave forwarding shims.
2. Keep tuning files with the GEMM that loads them and preserve relative lookup
behavior. Check wheel/package inclusion as well as source-tree execution.
3. Preserve module singleton state: every runtime caller must import the same
new communication adapter, not a second copy of its buffer registry.
4. Separate relocation commits from semantic changes. Use
`mechanical-refactor-verify` to reproduce moves, and compare test inventories
and CI registration coverage before/after. Only delete an experimental path
after checking its call sites, flags, source, and dedicated tests together.
5. Run the namespace/dispatch CPU tests, the registered-test validation hook,
pre-commit, and relevant GPU tests when available. State exactly which GPU
checks ran; import/AST checks do not prove numerical or performance parity.
Do not infer violations solely from a model name inside a group, a missing
`_jit_` prefix, use of a runtime utility, or absence of `BaseFusedOp` inheritance.
@@ -46,8 +46,8 @@ in-flight row as shipped.
| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
| --- | --- | --- | --- | --- |
| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
@@ -70,8 +70,8 @@ in-flight row as shipped.
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
@@ -148,8 +148,8 @@ Stable entries should be folded into the mainline family rows above.
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
@@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package.
| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
@@ -391,7 +391,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
candidate_path=(
"python/sglang/srt/layers/flashinfer_comm_fusion.py"
"<br>python/sglang/srt/layers/layernorm.py"
"<br>python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
),
active_keywords=(
"allreduce_fusion",
@@ -419,7 +419,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
pattern="AITER allreduce fusion",
candidate_path=(
"python/sglang/srt/distributed/communication_op.py"
"<br>python/sglang/srt/layers/communicator.py"
"<br>python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/layers/layernorm.py"
),
active_keywords=(
@@ -721,7 +721,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
FusionPatternSpec(
pattern="DeepSeek comm-prep fused RMSNorm + quant / flatten-quant",
candidate_path=(
"python/sglang/srt/layers/communicator.py"
"python/sglang/srt/layers/layer_boundary/residual/add_norm.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
"forward_mla.py"
"<br>python/sglang/srt/models/deepseek_common/attention_forward_methods/"
@@ -0,0 +1,249 @@
---
name: processor-model-parity
description: Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
---
# Processor model parity
`rust/sglang-processor` renders chat requests for SGLang's Rust hosts (the Rust
server, sgl-router, the renderer). A model is supported only when, for the same
request body, the processor produces **the same prompt text and the same prompt
token ids** as Python's `OpenAIServingChat`. Python is the reference, Dynamo is
the implementation, and SGLang code exists only where the two differ.
Scope: the request -> prompt -> token ids path, plus the checks that let hosts
serve the model's OpenAI requests through `/generate` (see "OpenAI layer").
Multimodal preprocessing is not covered here: `render_request` fails on media
content parts, so hosts send those chats to the engine. The fixture generator
serves every checkpoint text-only, so a media case records that rejection.
## Done means
- **All text should work**: every text chat the engine accepts renders and
tokenizes in the processor (sgl-router's `AllText`). Only what SGLang rejects or
the processor cannot reproduce goes to the engine; list it in the PR.
- **Chat and completions go through the engine's `/generate`**, with the router
rendering, tokenizing and parsing (`lower_chat`, `lower_completion`); the
engine's OpenAI route is only the `Unsupported` fallback. That needs the
cookbook's `--reasoning-parser` and `--tool-call-parser` ported. Verify the route
against a live engine.
The worked example is DeepSeek-V4-Flash-0731: `src/render/models/deepseek_v4.rs`
plus `tests/fixtures/parity/deepseek-v4-flash-0731.json`. Paths below are relative
to `rust/sglang-processor/` unless they start with `python/`.
## Where a model's code goes
| Piece | File | Needed when |
|---|---|---|
| Identity -> formatter | `src/render/selection.rs` (`select_chat_formatter`) | always; mirror the order of `chat_encoding.resolve_chat_encoding_spec` |
| Formatter variant | `src/render/mod.rs` (`ChatFormatter`, its `render_request` arm, plus the `render_prompt`, `stop_strs` and `resolve_thinking` arms) | always |
| SGLang-specific steps | `src/render/models/<model>.rs`, re-exported in `models/mod.rs` | only where Dynamo diverges from Python |
| Parity cases | `tests/fixtures/parity/<model-id>.json` | always |
| README row | the `render` table in `README.md` | always |
No new test code: `tests/parity.rs` checks every fixture in the directory.
`render_request(Value, ..)` takes the SGLang request **body**, not Dynamo's
`OAIChatLikeRequest`, because Python reads fields the trait lacks (`task`,
`continue_final_message`, the `reasoning` object). Pick the shape by how Python
renders the model:
- **Python calls its own encoder** (a non-`None` spec such as `dsv4` or `dsv32`):
give the model a dedicated `ChatFormatter` variant and `render_request` arm, like
DeepSeek-V4. If `selection.rs` already wires the model to a Dynamo
`PromptFormatter`, replace that branch.
- **Python calls `apply_chat_template`** (Jinja, spec `None`): give the model a
variant holding Dynamo's Jinja `PromptFormatter`, picked in
`ChatFormatter::from_tokenizer_config` from the template text (as
`parser/template_detection.py` picks it, where it has a detector), and build the
template context that `_process_messages` and `_apply_jinja_template` build, like
GLM-5.3 (`models/glm53.rs`) and Qwen3.8 (`models/qwen38.rs`). Dynamo's formatter
rewrites shapes its load-time probes find the template rejects (a later system
turn, for one); SGLang raises on them, so check those before rendering.
## Method
0. **Pick the checkpoint.** Use the one the model's cookbook page recommends
(`docs/cookbook/`), pin its Hugging Face commit as `revision`, and cache it.
Note whether its tokenizer ships a `chat_template`; that can change the spec.
1. **Trace the Python path for the model.** Write down every transformation
between the HTTP body and `prompt_ids`, in order:
- `protocol.py::ChatCompletionRequest.normalize_reasoning_inputs`: `reasoning`
and `reasoning_effort` set the default `thinking` / `enable_thinking`.
Reuse `render/reasoning.rs` for this rather than porting it again.
- `serving_chat._convert_to_internal_request`: `chat_template_kwargs.reasoning_effort`
replaces the request effort; server `--default-chat-template-kwargs` merge in.
- `chat_encoding.resolve_chat_encoding_spec`: which branch renders the model
(a spec such as `dsv4`, `dsv32`, `kimi_k3`, `inkling`, or `None` for Jinja).
It also reads the server's `--tool-call-parser` and whether the tokenizer has
a chat template. If the deployment depends on the parser, set
`tool_call_parser` in the fixture, and add it to `ChatFormatterOptions` in the
same PR. Note any per-checkpoint state the server resolves once (DeepSeek-V4's
effort profile reads `encoding/encoding_dsv4.py`).
- That spec's branch in `serving_chat._apply_jinja_template` / `_encode_messages`:
message dumping, content flattening, `_handle_last_assistant_message`, system
insertion, tools, model-specific fields, the encoder or `apply_chat_template`
call, and how the prompt is tokenized.
- The encoder itself (for example `encoding_dsv4.py`), line by line. History
rules live there: dropped turns, `</think>` on empty reasoning, argument
formatting, and the errors it raises.
This list is the **divergence inventory**.
2. **Find Dynamo's counterpart.** Check `dynamo_renderer::native_formatter_for`,
the model's low-level encoder (for example `deepseek::v4::encode_messages_with_options`),
or the Jinja template. Prefer the lowest-level Dynamo entry that lets SGLang own
request semantics. Dynamo's OpenAI-level formatters apply their own defaults
(they filter tools by `tool_choice`, inject `response_format`, and map effort),
which often differ from SGLang's. Diff the low-level encoder against Python's
for the rules in step 1.
3. **Write one request case per inventory item** in the fixture (`name` +
`request` only), then generate the expected outputs from Python (below). Cover
the checklist; reuse the request shapes from the DeepSeek-V4 fixture where they apply.
4. **Implement.** Start from a plain Dynamo call and run `tests/parity.rs`. For
each failing case, add the smallest SGLang step that fixes it, with a one-line
doc comment naming the Python source it mirrors. Never patch the fixture to
match Rust.
5. **Verify** text offline, then token ids with the checkpoint cached (commands
below). If sgl-router or the renderer uses the formatter, run their suites too.
## Checklist (DeepSeek-V4 case that covers each)
- **Message dump**: roles lowercased; unknown and null fields dropped; `user`
reduced to role and content; null content becomes `""`; history tool calls keep
only `ToolCall`'s fields. (`ignored_fields`, `tool_results`; GLM-5.3's `tool_call_extra_fields`)
- **Content parts**: which part types survive and the join separator. (`parts`)
- **Tools**: all request tools or only those `tool_choice` allows; field order
and defaults of `Function.model_dump`; message-level tools; empty lists; schema
types `_validate_request` rewrites (`varchar` becomes `string`). (`tools`,
`tools_none`, `tools_named`, `message_tools`, `empty_message_tools`; GLM-5.3's
`tool_schema_type_aliases`)
- **Template kwargs**: check reserved names (`apply_chat_template`'s own arguments)
after the server's defaults merge in. (GLM-5.3's `kwargs_tokenize`, `default_tokenize`)
- **Pydantic coercion**: typed request fields are coerced before rendering. For
example, `strict: 1` and `defer_loading: "false"` become booleans, and
`strict: null` is rejected. Probe the pydantic model in Python to get its exact
rules; don't guess them. (`tool_bool_coercion`, `tool_strict_null`, `continuation_coerced`)
- **Tool-call arguments**: how the encoder wants them. DeepSeek-V4 takes a compact
JSON string; other encoders format them their own way. Keep key order
(`serde_json` `preserve_order` is declared in `Cargo.toml`). Floats must print as
Python's `json.dumps` prints them (`1e-06`, `1e+16`, `100.0`).
(`tool_results`, `agentic_thinking`, `tool_float_values`, `history_float_arguments`)
- **Thinking**: kwargs `thinking` > request effort (`!= "none"`) > `reasoning.enabled`
> server default kwargs > `SGLANG_DEFAULT_THINKING`. `enabled` follows Python
truthiness (`1` is on); strings use Python's yes-word set.
(`thinking_false`, `thinking_none`, `drop_thinking_ignored`, `reasoning_enabled_numeric`, `reasoning_enable_string`)
- **Effort**: precedence between kwargs, `reasoning`, `reasoning_effort` and env;
the checkpoint's profile mapping. (`effort_high`, `effort_max`, `kwargs_effort`, `effort_conflict`, `reasoning_object`)
- **Final assistant turn**: becomes a user turn, or with `continue_final_message`
a separately tokenized prefix. Check whether content is flattened before the
split (it is for DeepSeek-V4). Run each check where Python runs it: a final
assistant turn's tool calls are discarded, so object checks on their arguments
belong after the split. Hosts reach the model through `render_prompt` too, so
check their continuation handling as well.
(`final_assistant`, `continuation_parts`, `continuation_bos`, `continuation_only`, `final_tool_call_no_arguments`, `continuation_tool_call_null_arguments`)
- **Model-specific fields and turns**: `task` placement, a system turn mid-conversation,
inserted empty system turns. (`task_action`, `task_after_developer`, `consecutive_task`, `mid_system`)
- **History**: the encoder's rules for earlier turns: reasoning kept or dropped,
whole turns dropped, closing tags on empty reasoning. (`multi_turn`, `thinking_multi_turn`)
- **Tokens**: Python's `tokenizer.encode` adds special tokens by default; the
continuation prefix is encoded alone and loses a leading BOS
(`_append_assistant_prefix_to_prompt_ids`). `tests/parity.rs` does both, using the
fixture's `bos_token_id`; `render_prompt` returns the prefix as its own segment so
hosts keep that boundary. (`continuation_bos`, `continuation_after_text`)
- **Errors**: reject what Python rejects. A case Python raises on is recorded with
`error`, and `tests/parity.rs` then requires `render_request` to fail. That
includes pydantic rejections and Python crashes (a 500 is still a rejection).
(`invalid_tool_arguments`, `continuation_only`) If Dynamo rejects a shape Python accepts, say in the
PR that hosts fall back to the engine for it.
- **Known gaps**: when a mismatch is inside Dynamo and out of the processor's
reach, fix it upstream and mark the case `known_gap` with the reason. The test
reports the case without failing, and fails once it matches so the marker gets
removed. Check a newer Dynamo release first. (`tool_float_values`: Dynamo's `deepseek::common::to_json`)
## Generating fixtures
`tests/scripts/generate_parity.py` loads the pinned snapshot and resolves the
spec and per-checkpoint state with SGLang's own resolvers. It then validates every
case with `OpenAIServingChat._validate_request`, which rewrites tool schemas, and
renders it through the real `_process_messages`, under the case's
`default_chat_template_kwargs` (the server's `--default-chat-template-kwargs`), and
writes:
- `prompt`, `token_count` and `token_sha256` for each case, or `error` when
Python raises (`name`, `request`, `default_chat_template_kwargs` and `known_gap`
are kept as written). An `AttributeError` aborts instead: the stub server lacks
something, so extend it;
- the fixture-level `config` (the `config.json` fields the processor reads, plus
resolved overrides such as the effort profile), `bos_token_id`, and for a Jinja
checkpoint its `chat_template`, so the test renders offline.
```sh
cd rust/sglang-processor
PYTHONPATH=$REPO/python HF_HUB_CACHE=<hub cache> HF_HUB_OFFLINE=1 \
python tests/scripts/generate_parity.py tests/fixtures/parity/<model-id>.json
```
When a new spec needs more, extend `serving_chat()` in the script (one place):
- **More server state:** set the extra attribute that `__init__` resolves, for
example `_dsv41_default_reasoning_effort`.
A spec that tokenizes inside `apply_chat_template(tokenize=True)` (Kimi K3) is
recorded by the template's text output. Its `render_request` returns the
segments the checkpoint encodes, and `tests/parity.rs` encodes them one by one.
Pin `revision` to a commit hash and regenerate only on purpose.
## Verify
```sh
cd rust
cargo test -p sglang-processor --locked # text, offline
HF_HUB_CACHE=<hub cache> cargo test -p sglang-processor --test parity --locked # + token ids
cargo clippy -p sglang-processor --all-targets --locked -- -D warnings
cargo check -p sglang-processor --no-default-features --features render,tokenizer --locked
```
The token check loads the pinned commit's snapshot from the cache. When the
snapshot is missing, the test prints `token ids not checked` (visible with
`-- --nocapture`), so confirm that line is absent before claiming token parity.
With the snapshot cached, the test also checks that the checkpoint resolves the
fixture's recorded DeepSeek-V4 profile.
## OpenAI layer
A host serves a model's OpenAI requests through `/generate` (`src/openai/`) only
once its render parity holds. Two more fixtures then cover the rest of Python's
OpenAI layer:
- `tests/fixtures/reasoning_parity/<parser>.json`, from
`tests/scripts/generate_reasoning_parity.py`: SGLang's `ReasoningParser` on
chunked outputs. A parser is served once `src/think/models/` ports it.
- `tests/fixtures/tool_parity/<parser>.json`, from
`tests/scripts/generate_tool_parity.py`: SGLang's `FunctionCallParser` on
chunked outputs, replayed by the test in `src/tool_call/models/mod.rs`. A tool
parser is served once `src/tool_call/models/` ports it.
- `tests/fixtures/openai_parity/<model-id>.json`, from
`tests/scripts/generate_openai_parity.py --model <dir> --engine-url <engine>`:
the `/generate` body `OpenAIServingChat` builds for each case, a live engine's
output for it, and the response Python builds from that output. Launch the
engine with the model's cookbook flags, including its parsers. Without the
weights, mark cases `synthetic` and write the engine output by hand, as GLM-5.3's
fixture does.
## Pitfalls
- Dynamo version bumps change rendering silently: rerun every fixture after bumping
`dynamo-renderer` or `dynamo-tokenizers`.
- `/generate` accepts only the API's sampling fields (`proto/sglang/api/v1/sampling.proto`).
A field Python sets in-process, such as glm47's `ebnf_full_assistant`, cannot be
sent, so such requests stay on the engine's route. Fixtures don't show this; a
live engine does.
- Python mutates the request before rendering (`_validate_request` rewrites tool
schemas), so fixture generators drive the server's own entry points.
- Env vars (`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`,
`SGLANG_DSV41_REASONING_EFFORT`) are read per request, as in Python. The
generator pins them and `tests/parity.rs` clears them, so fixtures do not
depend on the machine.
- Typed hosts (the renderer's `OAIChatLikeRequest` path) cannot carry every field,
such as `task` and message-level `tools`. Parity is defined on `render_request`;
report host-adapter gaps separately rather than bending the model code.
- Keep model code in its own file: no model-specific branches in `render/mod.rs`
beyond the dispatch arm.

Some files were not shown because too many files have changed in this diff Show More