Commit Graph
14664 Commits
Author SHA1 Message Date
minke.yu b1b303c9f2 mooncake: support host-memory payloads with intra-node NVLink transport
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.

When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
  per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
  the receiver validates and writes the block into its buffer, relying on
  the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
2026-10-25 14:04:52 +00:00
Liangsheng Yin 7d996300b9 [Perf] dsv41: Use the fused ratio-2 compressor in the prefill graph (#46230) 2026-10-23 21:33:42 -07:00
Liangsheng Yin 54df642fe6 [Perf] dsv41: Translate SWA locations once per prefill graph step (#46229) 2026-10-23 21:33:15 -07:00
Liangsheng Yin 564aec61c8 [Perf] dsv41: Feed wo_b the fused MXFP8 attention output in the prefill graph (#46231) 2026-10-23 21:31:33 -07:00
Yihao WangandClaude Opus 5.5 a7183a8246 [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners (#45145)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:28:56 -07:00
Yihao WangandClaude Opus 5.5 af3f9e4c8f [Feat] spec: Let out-of-tree platforms provide their own speculative worker (#45144)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:24:58 -07:00
MickandMick Qian f09cec65e9 [Fix] diffusion/storage: Join uploads before request cleanup (#46068)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:19:10 +08:00
MickandMick Qian f4ecaae888 [Fix] diffusion/mesh: Guard asynchronous job publication and defaults (#46081)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:18:25 +08:00
MickandMick Qian cdab6c1bcc [Fix] diffusion/loader: Preserve checkpoints on read failures (#46095)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:17:46 +08:00
f2c634961b [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop (#46105)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 12:17:04 +08:00
Liangsheng Yin 3b95120f00 [Fix] dsv41: Revert MegaGate routing in prefill (#46242) 2026-10-23 21:15:38 -07:00
55f23a8b59 [Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations (#46151)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:55:13 -07:00
Liangsheng Yin 3c6a6c8068 [Perf] dsv41: Run MegaGate routing in prefill (#46232) 2026-10-23 20:31:50 -07:00
Liangsheng Yin 17d92ea671 [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe (#46219) 2026-10-23 20:17:37 -07:00
Richard WangandMick Qian 7cefa60765 [Perf] diffusion: Wait on the socket when the scheduler is idle (#46007)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 11:02:24 +08:00
25ba24e9a7 [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma (#46109)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 11:00:14 +08:00
42ac81c655 [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell (#46107)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 10:59:25 +08:00
jambow0320andShangming Cai de7055f596 [Refactor] pd: Consolidate sender and receiver implementations (#41140)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-24 10:50:10 +08:00
jacky.cheng 88ac7c02a6 [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention (#45838) 2026-10-23 19:42:54 -07:00
Jackey Huaandhnyls2002 a77998e157 [Feat] openai: parse DeepSeek output through deepseek-recipe (#45907)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 19:23:22 -07:00
Liangsheng Yinandkpham-sgl 79701a2549 [Perf] dsv41: Read Engram rows from peer device shards (#46180)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-23 19:01:26 -07:00
Yihao WangandClaude Opus 5 327c906764 [Perf] [XPU] diffusion: Avoid QKV copies in flash attention (#41944)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
Yihao WangandClaude Opus 5 5bef70c0a2 [Perf] flux2: Build the single-block output projection without a cat (#41943)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
MickandMick Qian 1d4dcc81cc [Perf] diffusion/registry: Reduce model discovery work at startup (#46043)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 09:30:04 +08:00
47c0791ad1 [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters (#46099)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:28:55 +08:00
bbc77e7db4 [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) (#46110)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:27:39 +08:00
4e2d5b17ad [Perf] diffusion: Keep every component resident in auto mode when memory allows (#45978)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:26:13 +08:00
be8600d02d [Fix] diffusion: Step MOVA latents outside the serial-CFG branch (#45993)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:54 +08:00
a08bb0ec9e [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT (#46108)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:08 +08:00
e66c9e981d [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving (#45977)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:22:06 +08:00
2e6d87e2ca [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA (#46102)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:20:32 +08:00
386fa9ecfd [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality (#46106)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:19:12 +08:00
5b8955e70f [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses (#45965)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:17:41 +08:00
738e4535ea [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps (#46101)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:14:10 +08:00
3712b5944d [Perf] diffusion: Patchify Wan latents straight into tokens (#46148)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:12:49 +08:00
MickandMick Qian 2d5ad3243f [Fix] diffusion/http: Allow bounded graceful shutdown (#46169)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 09:11:15 +08:00
Jackey Huaandhnyls2002 e22a530859 [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe (#45315)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 18:05:35 -07:00
Khoa PhamandClaude Fable 5.1 ee2dd727a3 [Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate (#46163)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-23 17:54:03 -07:00
Liangsheng Yin 79e5c4bb9b [Perf] dsv41: Trim per-step DSpark decode overhead (#46176) 2026-10-23 17:24:06 -07:00
Qiaolin Yu 5994c5597d [Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation (#41169) 2026-10-23 17:15:21 -07:00
Baizhou Zhangandsid-rp 6a319e7b34 [Fix] muse: Support Responses namespace tools and defer_loading (#44685)
Co-authored-by: sid-rp <sidrp@meta.com>
2026-10-23 16:48:08 -07:00
0d592c475b [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill (#46066)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-23 16:47:14 -07:00
Cheng Wan 855c092deb [Fix] inkling: Keep BF16 LM-head operands with FP32 output (#45335) 2026-10-23 15:51:44 -07:00
Liangsheng Yin f05ad5ec01 [Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel (#46167) 2026-10-23 15:06:09 -07:00
Rohan VarmaandJun Wan 15dc838f9b [Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections (#45760)
Signed-off-by: Rohan Varma <rohanv@nvidia.com>
Co-authored-by: Jun Wan <8323392+junwan01@users.noreply.github.com>
2026-10-23 14:43:54 -07:00
Baizhou Zhang ea226cf782 [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2 (#43774) 2026-10-23 14:21:31 -07:00
Yuhao YangandClaude Opus 5.5 5f7542d762 [Perf] dsv41: Build the MegaGate row masks once per forward (#46071)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 14:15:01 -07:00
27ac614f05 [Feat] parser: Add checkpoint-driven response-template adapters (#40479)
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-23 14:05:54 -07:00
em-dreyer 4542fc8f59 [Fix] mistral3: Subclass LlavaForConditionalGeneration instead of wrapping it (#45656) 2026-10-23 14:05:32 -07:00
Michael 194abb8e54 [CI] [AMD] diffusion: Drain the Cosmos Dreams decode stream after freeing its outputs (#45983) 2026-10-23 13:26:56 -07:00