13347 Commits
Author SHA1 Message Date
BingjiaWang 4e60127fb4 [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (#40323) 2026-10-20 18:27:25 +08:00
Siyuan Chen e9fdf3ad0a [DSV4.1][PD][5/N] Decode node support DP attention (#40177) 2026-10-20 18:23:37 +08:00
minke.yu 2087760ea8 fix(v41): materialize CP request rows in select_compact_prefill
PR #42922 covered prefill_requests but the compact-candidate consume path
(DSpark draft consuming target candidates) still subscripted req_rows
directly, crashing with TypeError under CP8+DSpark5 prefill when
out_page_indices is set. Route through materialize_prefill_request_rows;
non-CP behavior unchanged (returns req_rows as-is).
2026-10-20 12:10:08 +08:00
minke.yu ef1a5955f3 PR #39306: H20-3e block-FP8 GEMM configs + adapt tables to NVIDIA H20
Net diff of PR #39306 vs upstream/main (kernel path already landed via #39657).
Also duplicated the six #41098 quantization tables and the six #39306 gemm
tables under device_name=NVIDIA_H20: config lookup is exact-filename and our
GPUs report 'NVIDIA H20' (96GB), not H20-3e. H20-3e shares the SM90 compute
configuration, so the tuned tables transfer.
2026-10-20 10:26:58 +08:00
Yuhao Yang 7e17dee8b7 Keep wo_a small-batch on its validated G=2 shape 2026-10-20 10:24:26 +08:00
Yuhao Yang 0090df5d64 Gate the G=1 wo_a GEMV to sm90 2026-10-20 10:24:26 +08:00
Yuhao Yang 4938eea2dc [DSV4.1] Enable Hopper decode optimizations for TP8 2026-10-20 10:24:25 +08:00
冯骐 8c6cbd8c40 Support DSpark decoder bounded replay with DSV4.1 vision CP
Use tail-aware CP gathers and logits metadata, preserve original CP ownership of canonical routing IDs, and pass global token indices to DSpark. Adapt mHC state slicing and register CPU regression tests. Based on PR40219; tail handling informed by PR40161.
2026-10-20 10:20:57 +08:00
冯骐 c82aed9fef Fix DSV4.1 Torch prefill indexer request-pool rows under CP
Materialize actual request-pool IDs from per-rank real row counts. Preserve zero-row requests, bounded-replay tails and the non-CP fast path. Add CPU regression coverage.
2026-10-20 10:16:54 +08:00
minke.yu e6f6420275 fix(v41): import Backend/Phase/with_phase in vision prefill CP validator 2026-10-20 00:43:11 +08:00
minke.yu 566176d0cf deps: pin sglang-kernel back to 0.4.7 for torch 2.13 ABI
da343ba5d1 (#42604 cherry-pick) bumped sglang-kernel to 0.4.9, whose
wheels are compiled against torch 2.14 (requires_dist torch==2.14.1 and
references c10::cuda::CUDAErrorLogCapture, absent in torch 2.13).
Import died with undefined symbol on this torch-2.13 branch.
sgl-deep-gemm 0.2.2 has no torch pin and stays.
2026-10-19 23:02:11 +08:00
minke.yu 4f52678224 fix(mm_schedule): _collect_image_span_requests emit 4-tuple overlapping
The vision-CP cherry-pick stored (idx, item, token_count) 3-tuples into
PerImageRequestInfo.overlapping, but the field and its consumers
(_assemble_per_image_chunk, phase-1a) use (idx, item, start, end) with
cache key (hash, end-start+1). First owner-mode image request died with
'not enough values to unpack (expected 4, got 3)'.
2026-10-19 21:47:57 +08:00
minke.yu 068acff09d fix(mm_schedule): port _item_overlap from upstream for vision-CP span collect
fd86688428 (vision CP single-owner encoding) cherry-pick used
_item_overlap in _collect_image_span_requests without the definition;
first multimodal prefill request killed every scheduler with NameError.
2026-10-19 21:21:33 +08:00
minke.yu f9056f0fcd fix(dsv4 fp4_indexer): 128B-align SM90 paged score rows for deep_select
fp4_index_logits_paged padded rows to 8 floats (32B) and
fp4_index_logits_candidates did not pad at all, but the tvm-ffi
deep_select topk kernel enforces kInputAlignmentBytes=128 on the input
row stride. bs>1 decode/CUDA-graph capture died with
'Tensor stride for dimension 0 is not aligned to 128 bytes'.
Candidates kernel now takes a separate OUT_STRIDE constexpr.
2026-10-19 20:55:01 +08:00
minke.yu 2aae0cb4ee fix(deep_gemm_wrapper): import DEEPGEMM_NEED_TMA_ALIGNED_SCALES/DEEPGEMM_SCALE_UE8M0 in compile_utils
Cherry-pick da343ba5d1 (#42604) lost the configurer import hunk; runtime
NameError killed every scheduler right after weight load on SM90.
2026-10-19 20:16:46 +08:00
minke.yu ea5fc832c1 scheduler: CP-symmetric idle check for health-check admission
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.

Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).

(cherry picked from commit b91137ab98)
2026-10-19 10:47:57 +08:00
minke.yu 9894dc789c G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
  pool_transfers (comp prepare_prefetch + build_hicache_transfers +
  sidecars) so DeepSeek V4's logical-anchor KV goes through
  batch_exists_v2 side-pool existence instead of v1 batch_exists which
  always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
  backend is attached (not only decode-hicache mode; SWA models forbid
  decode radix+hicache so P nodes could never query L3); construct
  RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
  disagg-prefill result path (it bypasses batch_result_processor, so
  drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.

(cherry picked from commit e7dc175e94)
2026-10-19 10:47:07 +08:00
minke.yu 77e7555ae3 G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。

(cherry picked from commit 71c6d8bc55)
2026-10-19 10:46:23 +08:00
minke.yu 319ee3c056 G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
  记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
  set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费

B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
  + query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
  _tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
  v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit

(cherry picked from commit 4eabf5e228)
2026-10-19 10:46:23 +08:00
minke.yu fd86688428 PR #40219: DeepSeek V4.1 vision CP and single-owner image encoding
Source: https://github.com/sgl-project/sglang/pull/40219 (applied via GitHub .diff, 3way)
Adaptations for v0.5.21 base:
- attn_dp_enabled_of(cfg) -> cfg.enable_dp_attention (#41818 not in base)
- dropped _validate_joint_schema_request call (#42721 Clef, not in base)
- moved Backend import before the TRT-LLM prefill-graph check
2026-10-19 10:45:10 +08:00
Vladislav Nosivskoy da343ba5d1 [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 (#42604)
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2ff3e6dce1)
2026-10-19 10:38:40 +08:00
minke.yu 9ea02a7201 PR #40421: reuse shared compressed KV dequantization in PD prefill
Source: https://github.com/sgl-project/sglang/pull/40421 (applied via GitHub .diff, 3way)
2026-10-19 10:37:44 +08:00
minke.yu 6c1c54b514 fixup: import is_gfx95_supported for #41294 and #40860 ROCm branches
dsv41_sparse.py and graph_variants.py referenced is_gfx95_supported without
importing it (latent NameError on the ROCm path).
2026-10-19 10:37:08 +08:00
minke.yu 51624199b4 PR #41294: use the fused low-ratio compressor on Hopper (sm90 gate)
Source: https://github.com/sgl-project/sglang/pull/41294 (applied via GitHub .diff, 3way)
Skipped: test_dsv41_fused_compress.py (file not in v0.5.21 base).
2026-10-19 10:35:29 +08:00
minke.yu 31b3758c99 PR #42365: fuse Hopper prefill indexer scoring into one Triton kernel
Source: https://github.com/sgl-project/sglang/pull/42365 (applied via GitHub .diff, 3way)
Conflicts resolved: unioned env flags with #40945 SM90 flags; merged fused
scoring loop with #40860 candidate_blocks path (candidate > fused > torch priority).
2026-10-19 10:33:13 +08:00
minke.yu 740372bef0 PR #40945: SM90 static verification indexing (grouped/length-aware indexer)
Source: https://github.com/sgl-project/sglang/pull/40945 (applied via GitHub .diff, 3way)
Adaptation: PR base predates #41659 which removed the Selection struct;
ported sm90_decode.py and call sites to the inputs.out_page_indices interface
(raw_indices=None on decode paths, reset via inputs.reset_outputs()).
2026-10-19 10:30:41 +08:00
minke.yu 19fe20baec PR #40860: SM90 candidate indexers (compact candidate dispatch)
Source: https://github.com/sgl-project/sglang/pull/40860 (applied via GitHub .diff, 3way)
Conflicts resolved: kept #41658 padded-return trim in fp4_index_logits_paged,
took PR side for candidate_blocks branches, SM90 graph gate, deep_select test param.
Skipped: test_dsv41_candidate_graph_variants.py (file not in v0.5.21 base).
2026-10-19 10:22:09 +08:00
yuyu5333 534842ac08 [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. (#41561)
Co-authored-by: Claude Code <noreply@anthropic.com>
(cherry picked from commit b50fdeb0ef)
2026-10-19 10:16:18 +08:00
Yuwei An 4dd95a8817 [DSV4.1] Reduce eager Engram extend preparation (#42159)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d434ea444d)
2026-10-19 10:16:08 +08:00
Yuwei An 656e00dcd3 [DSv4.1] mxfp8 dispatch cache (#42156)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
(cherry picked from commit 2d38934493)
2026-10-19 10:15:46 +08:00
DarkSharpness a4b1363bf4 [DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs (#41659)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 520d6d0229)
2026-10-19 10:15:46 +08:00
DarkSharpness ad2acaa2df [DSv4.1] Fold q_rope_store into fused_q_norm_rope (#41657)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 379ec90fb2)
2026-10-19 10:15:45 +08:00
Yuwei An ecaaa37cb0 [Fix][DSV4.1] SWA page size with bounded replay (#42128)
(cherry picked from commit ef867fa40d)
2026-10-19 10:14:30 +08:00
DarkSharpness 1b70fc192c [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2f6e0ec45d)
2026-10-19 10:14:29 +08:00
DarkSharpness 3ebbb2d536 [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 5916999afb)
2026-10-19 10:14:29 +08:00
DarkSharpness 5d52cccbee [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit d75d5b33f7)
2026-10-19 10:14:28 +08:00
minke.yu 74d03143e2 PR #41098: H20 block-FP8 tuned GEMM configs
Source: https://github.com/sgl-project/sglang/pull/41098 (applied via GitHub .diff, 3way)
2026-10-19 10:13:59 +08:00
Xiaoyu Zhang 2b192aade6 [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (#41251)
(cherry picked from commit 391e665dab)
2026-10-19 10:13:35 +08:00
minke.yu 9247c97919 PR #40919: fix dspark folded result buffer reuse race
Source: https://github.com/sgl-project/sglang/pull/40919 (applied via GitHub .diff, 3way)
2026-10-19 10:12:23 +08:00
minke.yu 5e7190b87a PR #40758: escape literal DSV4.1 image placeholder in user text
Source: https://github.com/sgl-project/sglang/pull/40758 (applied via GitHub .diff, 3way)
2026-10-19 10:12:16 +08:00
e00930c548 [Cherry-pick to release/v0.5.21] [Test] Demote PD test RDMA openability check to a warning (#41681) (#41802)
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 19:44:12 -07:00
Kangyan-ZhouandBaizhou Zhang d7ff74b984 [Cherry-pick to release/v0.5.21] [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL (#41777) (#41788)
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2026-09-29 17:30:38 -07:00
Jiacong Fang bd78095030 [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B (#41059) 2026-09-29 00:38:58 -07:00
MickandMick Qian dc1bd46802 [diffusion] feat: support bounded exact conditioning cache across native models (#40470)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-29 15:27:22 +08:00
eeecho c7be3e935b [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts (#34528) 2026-09-28 23:08:44 -07:00
Anupa SajikumarandMa Mingfei 2ff52e3ceb [Intel][XPU]Enable chunked prefill scnearios for XPU with UT (#33804)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:28:25 +08:00
9ae43b0eee [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path (#40828)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:16:03 +08:00
Cheng Wan 5781206833 [PD] Give FakeKVReceiver ensure_abort_notified (#41618) 2026-09-28 21:59:17 -07:00
Cheng Wan 6dad152490 [Refactor] Document layer boundary contracts and integration (#41557) 2026-09-28 20:18:27 -07:00
Cheng Wan de9f850b36 [Refactor] Rename the module to layer_boundary (#41555) 2026-09-28 20:17:50 -07:00