zicai.pu
0a56a5f6e7
fix(hicache): preserve sparse restore checkpoints under host pressure
...
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.
Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
2026-10-30 11:35:29 +08:00
minke.yu
7f6fe22445
mem_cache: fail-soft on SWA load-back gaps instead of asserting
...
With long prompts and L3 under pressure, a matched prefix can contain a node
whose SWA component has neither a device nor a host value (host tombstone
evicted between match and load-back). build_hicache_transfers asserted on
this and killed the prefill scheduler.
Add SWAComponent.has_load_back_gap() and have build_load_back_spec return an
empty transfer spec when a gap is found, so the caller backs off and the
request is recomputed. Measured on peak15-v6 (1M, cap 64): 0 scheduler
exceptions, hit 59.4%->94.0%, TTFT p95 47.0s->3.4s with the SWA prefix-match
gate disabled.
2026-10-26 13:23:17 +00:00
minke.yu
f6fc6ac3e7
mem_cache: allow disabling the DSv4.1 SWA prefix-match gate via env
...
The unified-cache SWA component gates prefix matching on a live SWA window,
so a long prefix only matches up to the last sliding window and the tail is
recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate
can be turned off for measurement; the SWA window is then rebuilt/recovered
downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix:
c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test
(cold vs hit) identical.
2026-10-26 04:06:25 +00:00
minke.yu
0a3c370451
mooncake: probe pointers via cuda.bindings instead of ctypes libcudart
...
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
2026-10-25 14:04:56 +00:00
minke.yu
b1b303c9f2
mooncake: support host-memory payloads with intra-node NVLink transport
...
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.
When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
the receiver validates and writes the block into its buffer, relying on
the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
2026-10-25 14:04:52 +00:00
Liangsheng Yin
7d996300b9
[Perf] dsv41: Use the fused ratio-2 compressor in the prefill graph ( #46230 )
2026-10-23 21:33:42 -07:00
Liangsheng Yin
54df642fe6
[Perf] dsv41: Translate SWA locations once per prefill graph step ( #46229 )
2026-10-23 21:33:15 -07:00
Liangsheng Yin
564aec61c8
[Perf] dsv41: Feed wo_b the fused MXFP8 attention output in the prefill graph ( #46231 )
2026-10-23 21:31:33 -07:00
Yihao Wang and Claude Opus 5.5
a7183a8246
[Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners ( #45145 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:28:56 -07:00
Yihao Wang and Claude Opus 5.5
af3f9e4c8f
[Feat] spec: Let out-of-tree platforms provide their own speculative worker ( #45144 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:24:58 -07:00
Mick and Mick Qian
f09cec65e9
[Fix] diffusion/storage: Join uploads before request cleanup ( #46068 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:19:10 +08:00
Mick and Mick Qian
f4ecaae888
[Fix] diffusion/mesh: Guard asynchronous job publication and defaults ( #46081 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:18:25 +08:00
Mick and Mick Qian
cdab6c1bcc
[Fix] diffusion/loader: Preserve checkpoints on read failures ( #46095 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:17:46 +08:00
f2c634961b
[Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop ( #46105 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 12:17:04 +08:00
Liangsheng Yin
3b95120f00
[Fix] dsv41: Revert MegaGate routing in prefill ( #46242 )
2026-10-23 21:15:38 -07:00
55f23a8b59
[Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations ( #46151 )
...
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 20:55:13 -07:00
Liangsheng Yin
3c6a6c8068
[Perf] dsv41: Run MegaGate routing in prefill ( #46232 )
2026-10-23 20:31:50 -07:00
Liangsheng Yin
17d92ea671
[Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe ( #46219 )
2026-10-23 20:17:37 -07:00
Richard Wang and Mick Qian
7cefa60765
[Perf] diffusion: Wait on the socket when the scheduler is idle ( #46007 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 11:02:24 +08:00
25ba24e9a7
[Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma ( #46109 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 11:00:14 +08:00
42ac81c655
[Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell ( #46107 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 10:59:25 +08:00
jambow0320 and Shangming Cai
de7055f596
[Refactor] pd: Consolidate sender and receiver implementations ( #41140 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-24 10:50:10 +08:00
jacky.cheng
88ac7c02a6
[Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention ( #45838 )
2026-10-23 19:42:54 -07:00
Jackey Hua and hnyls2002
a77998e157
[Feat] openai: parse DeepSeek output through deepseek-recipe ( #45907 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-23 19:23:22 -07:00
Liangsheng Yin and kpham-sgl
79701a2549
[Perf] dsv41: Read Engram rows from peer device shards ( #46180 )
...
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
2026-10-23 19:01:26 -07:00
Yihao Wang and Claude Opus 5
327c906764
[Perf] [XPU] diffusion: Avoid QKV copies in flash attention ( #41944 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-24 09:33:26 +08:00
Yihao Wang and Claude Opus 5
5bef70c0a2
[Perf] flux2: Build the single-block output projection without a cat ( #41943 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-24 09:33:26 +08:00
Mick and Mick Qian
1d4dcc81cc
[Perf] diffusion/registry: Reduce model discovery work at startup ( #46043 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 09:30:04 +08:00
47c0791ad1
[Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters ( #46099 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:28:55 +08:00
bbc77e7db4
[Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) ( #46110 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:27:39 +08:00
4e2d5b17ad
[Perf] diffusion: Keep every component resident in auto mode when memory allows ( #45978 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:26:13 +08:00
be8600d02d
[Fix] diffusion: Step MOVA latents outside the serial-CFG branch ( #45993 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:23:54 +08:00
a08bb0ec9e
[Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT ( #46108 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:23:08 +08:00
e66c9e981d
[Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving ( #45977 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:22:06 +08:00
2e6d87e2ca
[Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA ( #46102 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:20:32 +08:00
386fa9ecfd
[Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality ( #46106 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:19:12 +08:00
5b8955e70f
[Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses ( #45965 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:17:41 +08:00
738e4535ea
[Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps ( #46101 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:14:10 +08:00
3712b5944d
[Perf] diffusion: Patchify Wan latents straight into tokens ( #46148 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:12:49 +08:00
Mick and Mick Qian
2d5ad3243f
[Fix] diffusion/http: Allow bounded graceful shutdown ( #46169 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 09:11:15 +08:00
Jackey Hua and hnyls2002
e22a530859
[Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe ( #45315 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-23 18:05:35 -07:00
Khoa Pham and Claude Fable 5.1
ee2dd727a3
[Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate ( #46163 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-23 17:54:03 -07:00
Liangsheng Yin
79e5c4bb9b
[Perf] dsv41: Trim per-step DSpark decode overhead ( #46176 )
2026-10-23 17:24:06 -07:00
Qiaolin Yu
5994c5597d
[Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation ( #41169 )
2026-10-23 17:15:21 -07:00
Baizhou Zhang and sid-rp
6a319e7b34
[Fix] muse: Support Responses namespace tools and defer_loading ( #44685 )
...
Co-authored-by: sid-rp <sidrp@meta.com >
2026-10-23 16:48:08 -07:00
0d592c475b
[Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill ( #46066 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-23 16:47:14 -07:00
Cheng Wan
855c092deb
[Fix] inkling: Keep BF16 LM-head operands with FP32 output ( #45335 )
2026-10-23 15:51:44 -07:00
Liangsheng Yin
f05ad5ec01
[Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel ( #46167 )
2026-10-23 15:06:09 -07:00
Rohan Varma and Jun Wan
15dc838f9b
[Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections ( #45760 )
...
Signed-off-by: Rohan Varma <rohanv@nvidia.com >
Co-authored-by: Jun Wan <8323392+junwan01@users.noreply.github.com >
2026-10-23 14:43:54 -07:00
Baizhou Zhang
ea226cf782
[Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2 ( #43774 )
2026-10-23 14:21:31 -07:00