Commit Graph
5109 Commits
Author SHA1 Message Date
minke.yu baba18bc7d hicache L3 prefetch: io_aux 传输并行化(按 op 分 lane)
build-sglang-image / build (push) Failing after 48m34s
打点实测单条 prefetch_io_aux_func 线程串行传输是所有请求的瓶颈:
mc8 下每请求 xfer 排队 0->1.9s 线性增长(每请求 5 批 x ~68ms 固定开销,
介质无关)。本提交把传输层拆成 N 条并行 lane(默认 4,
SGLANG_HICACHE_PREFETCH_IO_THREADS 可调):

- dispatch_prefetch() 用 crc32(request_id) % N 分 lane(内置 hash 对 str
  每进程加盐,跨 rank 不一致会导致 gloo 集合通信错位,禁用)
- 同一 op 的所有 chunk 保持在同一 lane,保序 reduce 语义不变
- 每 lane 独立 io 队列 + io_aux 线程 + sync 队列 + sync 线程 +
  独立 gloo completion 组(集合通信按 lane 过滤后的分发顺序保序)
- query 阶段的 hit_reduce 仍在单 prefetch_thread,不动
- 生产点(hiradix/unified/hybrid pp-ticket)改调 dispatch_prefetch
- 模拟器与两个单测文件同步适配

注意:SGLANG_HICACHE_PREFETCH_IO_THREADS 必须在实例内所有
TP/CP/PP rank 保持一致,否则 lane 组上的集合通信会错位挂死。
2026-10-10 19:04:49 +08:00
minke.yu a42ab6fce0 测试适配 #38348 新语义:SWA 双层缺失的 backuped 节点现在是合法匹配边界
build-sglang-image / build (push) Failing after 1s
旧语义(host_hit=0 + branching=9 + 重插建窗)被 fail-soft 取代:
FULL 段直接 host 命中(host_hit=leaf_len、branching=None),load-back
落 FULL 后由续写请求的 insert overlap 从 Full KV 重建窗口 SWA。
重写 test_swa_branching_seqlen_uses_host_full_hit 验证新链路:
match 命中 → load-back → 插入扩展 token 触发 overlap 重建 → rematch
设备全命中且洞口节点的窗口 SWA 已恢复。
2026-10-10 07:42:44 +08:00
minke.yu fa8ad18b5c 手工移植上游 #38348:hicache SWA prefetch fail-soft 降级
build-sglang-image / build (push) Failing after 1s
SWA 池 L3 取回短缺时不再整批丢弃 prefetch 结果:保留已取回的 Full KV
前缀,仅丢弃失败的 SWA 池(释放其 host buffer、从 comp_xfers 摘除),
load-back 时经既有 SWARebuild/RecoverSWAWithLockedFull 路径从 Full KV
重建窗口。Mamba 等不可重建池短缺时仍走原 all-or-nothing 整批丢弃。

配套改动(同 PR):
- swa.py validator:SWA 双层缺失但节点已 backuped(Full KV 在 host)时
  放行作为匹配边界;LOAD_BACK 收集遇双层缺失空洞时 break 而非 assert。
- prefetch_from_storage:用完整匹配前缀重算 last_hash,修复 anchor 节点
  无 host_value(异步 backup 未完成/降级嫁接)时哈希链锚错位。
- 适配差异:本树有 sidecar_hash_values 重构,belief 自愈沿用
  invalidation_hashes + trimmed_pages 语义;ongoing_prefetch 以
  CacheRequestHandle 为键,comp_xfers 改为显式参数传入检查函数。
- 单测同步更新为 fail-soft 语义(TP 一致性降级、失败池释放无泄漏、
  LOAD_BACK 锚定空洞停止收集)。
2026-10-10 07:23:15 +08:00
minke.yu 7578cdd218 revert: #42035 continuous prefill input polling (CP rank collective mismatch deadlock)
build-sglang-image / build (push) Failing after 39m1s
compose 3p1d regression on 1f6ca2cf6: all 3 P nodes wedged with split-brain
CP ranks (one blocked in _sync_hicache_ready_counts all_reduce, partner in
request broadcast). Root cause: #42035 wraps get_next_disagg_prefill_batch_to_run
(which contains the hicache attn_cp_group all_reduce) in rank-local guards
(len(result_queue)<2, _deferred_input_requests, skip_batch_creation); any
cross-rank divergence in batch-None decisions misaligns collective pairing
permanently. Continuous polling requires attn_cp_size==1 anyway, so cp2
deployments get zero benefit. Reverts 93d610c1fd + test adaptations
6ba34507f1/bde1fcbf51/1f6ca2cf68 + deploy bump 236686454c.
Keeps #42194 (9ef65b51c2) and checkpoint_disagg_prefill shim (c4f3e3be33).
Evidence: D:\B300\investigations\pd-deadlock-20261009.md
2026-10-09 16:39:06 +08:00
minke.yu 1f6ca2cf68 test: adopt upstream #42035 auxiliary-output asserts now that checkpoint_disagg_prefill shim exists
build-sglang-image / build (push) Successful in 25m55s
2026-10-09 11:32:33 +08:00
minke.yu bde1fcbf51 test: tolerate chunked kwarg in #42035 polling test override (branch shim keeps chunked flag) 2026-10-09 11:25:05 +08:00
Liangsheng Yin 9ef65b51c2 [mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public (#42194)
build-sglang-image / build (push) Successful in 25m53s
(cherry picked from commit 07e1a4f082)
2026-10-09 11:18:17 +08:00
minke.yu 6ba34507f1 test: adapt #42035 polling test to pre-#41520 name maybe_cache_unfinished_req
build-sglang-image / build (push) Successful in 26m20s
2026-10-09 11:10:39 +08:00
metamergebot 93d610c1fd [PD] Keep ingesting requests while a prefill forward result is pending (#42035)
build-sglang-image / build (push) Successful in 26m6s
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
(cherry picked from commit 6acf2f4736)
2026-10-09 10:56:14 +08:00
metamergebot 75758cb0a0 [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails (#42420)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com>
(cherry picked from commit 4ab720e655)
2026-10-08 15:41:51 +08:00
metamergebot 60e80ac7d1 [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert (#42264)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
(cherry picked from commit 2bae12b9b3)
2026-10-08 15:37:43 +08:00
Sasha Sidorov 4ef799aaa6 [PD] Wait for prefill completion before Mooncake early KV transfer (#41395)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c683190b40)
2026-10-08 15:36:21 +08:00
Aurick Qiao 1be762368b [Disagg] Validate state strides before Mooncake transfers (#41607)
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com>
(cherry picked from commit 1a0250c5b6)
2026-10-08 15:35:05 +08:00
DarkSharpness 4792b61332 [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 5916999afb)
2026-10-08 15:32:48 +08:00
DarkSharpness 62f1ed099e [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit d75d5b33f7)
2026-10-08 15:32:47 +08:00
Yuwei An 842ad7c4d4 [Fix][DSV4.1] SWA page size with bounded replay (#42128)
(cherry picked from commit ef867fa40d)
2026-10-08 15:32:45 +08:00
Mohammad Miadh Angkad b605d2d370 [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer (#41987)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
(cherry picked from commit 42f2e4fcb7)
2026-10-08 15:32:44 +08:00
Xinyuan Tong 6833498646 model: prune comments and redundant tests in dsv41 vision CP 2026-09-23 14:36:01 +08:00
Xinyuan Tong b48e2cb1eb model: TP-wide single-owner image encoding for DeepSeek V4.1
ViT and Aligner are replicated per TP rank, encoding each image eight
times with TP8 on both CP1 and CP8. Elect one owner per image and use
ordered full-span broadcasts with a six-phase agreement protocol.
Both CP1 and CP8 benefit while local cache hits preserve collective order.
2026-09-23 14:36:01 +08:00
Xinyuan Tong bfeb7cd9b2 model: support DeepSeek V4.1 vision with interleave prefill CP
The CP runner bypassed the vision merge and used bare text embeddings.
Merge image features before sharding so request-global offsets stay valid.
Canonicalize model IDs separately to preserve scheduler hash IDs.
Keep unsupported combinations guarded and isolate embedding overrides
from multimodal prefills without starving queued FCFS requests.
2026-09-23 14:36:00 +08:00
Yuwei An ddf5207630 [Fix] Handle chunked paged MQA metadata in DSV4.1 eager forwards (#40637)
build-sglang-image / build (push) Successful in 32m15s
2026-09-23 13:35:55 +08:00
minke.yu b081dd3d23 Merge branch 'main' into dsv41-pd 2026-09-22 14:56:35 +08:00
Mohammad Miadh AngkadandMohammad Angkad 4c81cd1b09 [KDA] Fix missing beta sigmoid in PTX prefill (#40685)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-21 23:24:38 -07:00
Guangda LiuandGuangda Liu 04c0913434 [HiSparse] Add MHA hisparse support for MiniMax M3 (#31446)
Co-authored-by: Guangda Liu <bingps@users.noreply.github.com>
2026-09-22 13:28:03 +08:00
095e45100b [AMD] [GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950 (#38545)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 22:24:29 -07:00
YAMY 9b59fc5db5 [ModelOpt][PP] Keep BF16 shared experts out of the NVFP4 fusion so TP1 pipeline stages can load (#40628) 2026-09-21 21:45:58 -07:00
e1daf68304 [AMD] [GLM-5.3-Flash Day 0] Honor fused and per-expert names in quark exclude (#39317)
Co-authored-by: Yikai Zhang <ykzhang12@gmail.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 21:26:42 -07:00
Khoa PhamandQiaolin Yu 018b73c7a0 [PD] Pack draft KV head slices for DCP transfers (#40500)
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
2026-09-21 21:11:27 -07:00
b44e248682 [AMD] [GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950 (#38546)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 21:07:37 -07:00
jthomson04 15ba54bd5d perf(engine): avoid timed waits for Engine responses (#39486)
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
2026-09-21 20:42:03 -07:00
Cheng Wan 1d025491f3 [Test] Set DP size in the mocked Metal profiler test (#40667) 2026-09-21 19:57:52 -07:00
90cf471723 [AMD] [GLM-5.3-Flash Day 0] Support non-2048 top-k widths in the DSA page-table transform (#39340)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-21 19:50:17 -07:00
jacky.cheng bc30fa1759 [AMD][Fix] AgentX HIP TPOT regression when SGLANG_SIMULATE_ACC_LEN is set (#40598) 2026-09-21 19:16:15 -07:00
Cheng Wan 9eda772a21 [Test] Handle tied top-k indices in graph-pool logprob regression (#40661) 2026-09-21 19:02:43 -07:00
Mohammad Miadh Angkadandmmangkad e332e1b84e [Fix] Don't write conv state from the fused KDA verify kernel (#39524)
Co-authored-by: mmangkad <mohammad.angkad@radixark.ai>
2026-09-21 18:33:03 -07:00
Dayananda VandClaude Opus 5 35eb7cf8d6 [Intel][XPU][KVCanary] Enable KV Canary on Intel XPU (#33520)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 09:19:01 +08:00
Ma Mingfei 046cd6f4ea [XPU][ci]: disable XPU NIXL disaggregation test (#40540) 2026-09-22 09:02:22 +08:00
Khoa Pham c4d3770a68 [Kimi K3] Fix CUDA graph stream explosion (#40640) 2026-09-21 17:49:50 -07:00
Cheng Wan 98c8dee23b Fix lint failure from draft-decode window test location (#40654) 2026-09-21 17:41:00 -07:00
042b6a488f [AMD] [GLM-5.3-Flash Day 0] Enable zero-RoPE MHA prefill on ROCm (#39338)
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com>
2026-09-21 17:38:07 -07:00
avalliappan-nvidia 61d0cf2074 [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts (#32673) 2026-09-22 08:17:57 +08:00
Vedant V JhaveriandCopilot 9fdb71732a Avoid materializing GDN QKV tensors during target verification (#33778)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-21 17:04:18 -07:00
Cheng Wan 506698761d [unified-memory] Hierarchical cache for every unified pool shape (#37507) 2026-09-21 16:50:37 -07:00
Cheng Wan 22587fb15c [Fix] Run KV canary hooks for context-parallel prefill (#40642) 2026-09-21 16:46:22 -07:00
Yuxuan ZhangandXinyuan Tong 00986c81be Support GLM-5.3-Flash hybrid attention CPU offload and PD index mapping (#40310)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-09-21 16:03:03 -07:00
YAMY 0229025127 [Spec][PP] Launch extend microbatches before the spec output exchange (#40499) 2026-09-21 15:47:03 -07:00
Liangsheng Yin a5c2cc517c [CI] Split the CI control labels into four axes and resolve them live (#40527) 2026-09-21 15:37:28 -07:00
8bde82c0ad [AMD] [GLM-5.3-Flash Day 0] Build the fused DSA k-pool top-k JIT kernel on HIP (#39339)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-21 15:24:44 -07:00
Cheng Wan acac4dd9d9 [Refactor] Clean up parallel runtime comments (#40632) 2026-09-21 14:32:22 -07:00
jacky.cheng e0c2e8dc4d [AMD] Tune Qwen3.5 TP4 GDN recurrent launch on gfx950 (#39987) 2026-09-21 13:20:54 -07:00