minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
18,557 Commits 2 Branches 0 Tags
c74a4037fb0fb7afc6384c7b70b1f9773119548f
Commit Graph
12909 Commits
Author SHA1 Message Date
minke.yu c74a4037fb mooncake: probe pointers via cuda.bindings instead of ctypes libcudart 
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
 2026-09-21 12:46:25 +08:00
minke.yu b963295489 mooncake: support host-memory payloads with intra-node NVLink transport 
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.

When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
  per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
  the receiver validates and writes the block into its buffer, relying on
  the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
 2026-09-21 11:53:26 +08:00
abing fa826e08b1 Reuse shared compressed KV dequantization in DeepSeek V4.1 CP prefill 
(cherry picked from commit 44eb378e94610e4ccc6b990bf09c105f8c8f9ee9)
 2026-09-20 22:12:47 +08:00
SYChen123 3810f531a8 Support V4.1 decode vision MegaMoE 
(cherry picked from commit 98aa79972d6d31f30a3c5bc83b9d7e3009ae638a)
 2026-09-20 22:12:43 +08:00
SYChen123 2580c24d1b Support V4.1 DP attention in DP-only DSpark PD 
(cherry picked from commit 6ef0ebba730b3fffde4b59b953c5c9b58a35c4ed)
 2026-09-20 22:11:48 +08:00
abing 4f22146e51 Avoid host synchronization in DeepSeek V4.1 CP prefill 
(cherry picked from commit 510596019fd5782d494c9b5227bf784d46661f6f)
 2026-09-20 22:07:30 +08:00
abing 12e3b82e52 run pass cp+megamoe+bcg 
(cherry picked from commit 85b8a150ef69c35cbd8c3fe940e41b414b903757)
 2026-09-20 22:07:29 +08:00
abing 8305f66fc8 fix bug 
(cherry picked from commit 60d31cb0c268c4b733ab290e133e3763b65f090e)
 2026-09-20 22:07:24 +08:00
abing 21a4a16b4b update code 
(cherry picked from commit 1453150a0bdf2cce425e9e2da83ef16b8e8f2859)
 2026-09-20 22:07:24 +08:00
abing fc954b7e08 add test 
(cherry picked from commit 094abd5e90af77507acd7c91513f69b5b29b685b)
 2026-09-20 22:07:24 +08:00
abing c2059c4fb2 run pass llm cp 
(cherry picked from commit 1d85394563d96cf22d5d84f15c0e9043582419e4)
 2026-09-20 22:07:23 +08:00
HuangJi 2a0cb2f04e [Diffusion][MiniMax-H3] Add SM120 Sage compute for SubBlock sparse attention (#40116) 2026-09-20 16:40:28 +08:00
Shuwen WangandSeokhoon Kang 9f3d275940 [HiCache] Fix sparse hybrid transfer layer IDs (#37870) 
Co-authored-by: Seokhoon Kang <sh.kang@postech.ac.kr>
 2026-09-20 16:24:24 +08:00
amd-danli103andHAI e54009240a [AMD][DSV4] feat: enable DSpark with fp8 unified_kv on gfx950 (#38901) 
Co-authored-by: HAI <hixiao@gmail.com>
 2026-09-20 01:16:39 -07:00
Ke Bao a8a4d86be9 Remove swa and mamba radix cache (#40313) 2026-09-20 16:16:27 +08:00
Qiaolin Yu f4c256354c [kimi k3][pd disagg] support pp prefill + dcp decode with dspark (#40045) 2026-09-20 00:15:28 -07:00
Yuhan ChenYuhan ChenXiaoyu Zhang
 		99a44c88d4 Add out-of-tree DFlash extension points (#38740) 
Co-authored-by: Yuhan Chen <yuhanc@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
 2026-09-20 14:54:46 +08:00
Yuxuan ZhangandXinyuan Tong 9f21fbc34b [GLM-5.3-Flash] Reduce KPool planning synchronization and overlap indexer preparation (#39695) 
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-09-19 23:51:17 -07:00
Yuxuan ZhangandXinyuan Tong c8eb54c41d Fuse GLM-5.3-Flash KDA projections and prefill metadata (#39688) 
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2026-09-19 23:46:33 -07:00
chaijiacheng888 2d216a11f8 [Model] Serve DeepSeek-OCR-2 with its official 768px local-crop geometry (#38996) 2026-09-20 14:17:21 +08:00
MickandMick Qian 031bff5dd3 [diffusion] chore: batch qwen-image 2.1 targets and document measured deployment recipes (#40408) 
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
 2026-09-20 13:59:36 +08:00
Sasha Sidorov 59dd2fc734 [2/N] [Kernel] Fuse padding-preserving HiSparse slot translation (#39837) 2026-09-20 12:04:03 +08:00
Mohammad Miadh AngkadandMohammad Angkad df0dc44931 [Fix] Forward SWA prealloc reclaim through the DSV4 HiSparse allocator (#40354) 
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
 2026-09-19 20:27:51 -07:00
huangtingweiandChao Shi 020703923d [PP + HiCache] Add PP Prefetch Tickets for eager cross-stage storage prefetch (#36700) 
Co-authored-by: Chao Shi <stepinto@live.com>
 2026-09-20 11:19:31 +08:00
zijiexiaClaude Opus 5kpham-sgl
 		113f6f080e [PD] Enter the custom mem pool once when allocating DCP pack buffers (#40284) 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
 2026-09-19 19:34:58 -07:00
Tri Dao d82d653f96 Enable optimistic prefill for Mamba radix-cache models (#40184) 2026-09-19 19:28:11 -07:00
huangtingweiandZhangheng e9300f643e [Unified Cache][9/N] add opt-in MLA load deduplication for Mooncake Linker (#39565) 
Co-authored-by: Zhangheng <hzh0425@apache.org>
 2026-09-20 09:56:56 +08:00
AndyLi429andAndyLi429 d903351a66 [NPU][bugfix] update low latency quantization input and update MXFP8 tests (#38831) 
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com>
 2026-09-20 09:55:30 +08:00
MickMick QianBBuf
 		f9c2791460 [diffusion] model: support qwen-image-2.1 (#39983) 
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
 2026-09-20 09:46:09 +08:00
BourneSun0527andEven Zhou d2f291c934 [NPU][DSV4]dsv4 enable cpp (#39820) 
Co-authored-by: Even Zhou <even.y.zhou@outlook.com>
 2026-09-20 09:17:27 +08:00
YAMY 9cc7da2ab0 [MegaMoE] Wire Qwen MoE blocks to DeepGEMM MegaMoE (MXFP4 and NVFP4 experts) (#38080) 2026-09-19 16:00:38 -07:00
Po-Han Huang (NVIDIA)Julien LinYAMY1234Yangmin Li
 		3a64faa1f2 Fix disagg PP MTP for GLM-5.2 (#39378) 
Co-authored-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
 2026-09-19 13:55:07 -07:00
metamergebotpranjalsshJialin OuyangJialin OuyangLianmin Zheng
 		8139a1740e [Scheduler] Count complete prefill bursts and their tokens (#40006) 
Co-authored-by: pranjalssh <pranjalssh@fb.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Co-authored-by: Jialin Ouyang <jialino@meta.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2026-09-19 12:56:15 -07:00
Zhiqiang Xie 7a6c652c77 [HiCache] Auto-size the host pool to fit available host memory (#40135) 2026-09-19 12:50:43 -07:00
amd-danli103 2305242f51 [AMD][DSV4] fix: skip compressed-KV metadata on the draft worker in the HIP radix backend (#40205) 2026-09-19 12:13:49 -07:00
metamergebotandcctry 9e5a62a767 [Logprob] Serve input-logprob temporaries from CUDA-graph-pool dead space (#40038) 
Co-authored-by: cctry <csycfl@gmail.com>
 2026-09-19 12:03:52 -07:00
kkandwunhuang c5326d28a3 [AMD] dsv4: pick kv_splits per index stream, not by occupancy alone (#39968) 
Co-authored-by: wunhuang <wunhuang@amd.com>
 2026-09-19 11:53:54 -07:00
Junpan WuBaizhou ZhangYuwei An
 		7b67a96640 [DSV4] Chunk the indexer MQA logits by query rows under a free-memory budget (#39095) 
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
 2026-09-19 11:53:20 -07:00
Zhang, JiejingXiaobo ChenHAI
 		993d1fccba [ROCm] Widen the HiCache JIT copy rounds and enable the K-only host pool (#37152) 
Co-authored-by: Xiaobo Chen <xiaobche@smci355-ccs-aus-n05-33.prov.aus.ccs.cpe.ice.amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
 2026-09-19 08:58:10 -07:00
Xiaoyu Zhang 76f9213a41 [Fix] Keep mHC context out of non-V4 compiled MoE forwards (#40353) 2026-09-19 21:41:51 +08:00
Benjamin TruongandXiaoyu Zhang 83e29d6c5a [perf] Optimize w4a8 MoE for glm5.2 on H200 (#38220) 
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
 2026-09-19 20:34:25 +08:00
Xiaoyu Zhang 7fac84b639 [DSV4.1] Reduce mHC, metadata and small-batch router overhead (#39704) 2026-09-19 19:51:21 +08:00
DarkSharpnessandBBuf d1acbe0746 [DSV4.1] Big fused wo_a quant (#39957) 
Co-authored-by: BBuf <1182563586@qq.com>
 2026-09-19 19:46:53 +08:00
Xiaoyu Zhang cb22f2451e [Cleanup] Deduplicate kernel tests, diffusion fixtures and benchmark helpers (#40265) 2026-09-19 19:45:38 +08:00
Xiaoyu Zhang 0b0d2c257a [Fix] Repair CI fixtures and ROCm speculative tree device checks (#40325) 2026-09-19 18:16:05 +08:00
Cheng Wan 3a5f52e144 Record a process's placement at publish, not at group build (#40071) 2026-09-18 23:52:37 -07:00
Lianmin Zheng 5d703de9e4 [HiCache] Size MHA host pools from device row width (#40304) 2026-09-18 23:37:58 -07:00
Xiaoyu Zhang 6533223502 [Lint] Fix logits processor formatting on main (#40303) 2026-09-19 14:31:56 +08:00
Xiaozhu Mengandmxz 111aeedd37 [Runtime] Add decode CUDA graph hooks for eager logits processing (#40222) 
Co-authored-by: mxz <mxz@fb.com>
 2026-09-18 23:27:03 -07:00
Lianmin Zhengcctrycctry
 		6e1338dd1e Fix prefetch attempt cleanup on abort (#40262) 
Co-authored-by: cctry <cctry@meta.com>
Co-authored-by: cctry <cctry@fb.com>
 2026-09-18 22:46:17 -07:00