Commit Graph
13336 Commits
Author SHA1 Message Date
minke.yu 4f52678224 fix(mm_schedule): _collect_image_span_requests emit 4-tuple overlapping
The vision-CP cherry-pick stored (idx, item, token_count) 3-tuples into
PerImageRequestInfo.overlapping, but the field and its consumers
(_assemble_per_image_chunk, phase-1a) use (idx, item, start, end) with
cache key (hash, end-start+1). First owner-mode image request died with
'not enough values to unpack (expected 4, got 3)'.
2026-10-19 21:47:57 +08:00
minke.yu 068acff09d fix(mm_schedule): port _item_overlap from upstream for vision-CP span collect
fd86688428 (vision CP single-owner encoding) cherry-pick used
_item_overlap in _collect_image_span_requests without the definition;
first multimodal prefill request killed every scheduler with NameError.
2026-10-19 21:21:33 +08:00
minke.yu f9056f0fcd fix(dsv4 fp4_indexer): 128B-align SM90 paged score rows for deep_select
fp4_index_logits_paged padded rows to 8 floats (32B) and
fp4_index_logits_candidates did not pad at all, but the tvm-ffi
deep_select topk kernel enforces kInputAlignmentBytes=128 on the input
row stride. bs>1 decode/CUDA-graph capture died with
'Tensor stride for dimension 0 is not aligned to 128 bytes'.
Candidates kernel now takes a separate OUT_STRIDE constexpr.
2026-10-19 20:55:01 +08:00
minke.yu 2aae0cb4ee fix(deep_gemm_wrapper): import DEEPGEMM_NEED_TMA_ALIGNED_SCALES/DEEPGEMM_SCALE_UE8M0 in compile_utils
Cherry-pick da343ba5d1 (#42604) lost the configurer import hunk; runtime
NameError killed every scheduler right after weight load on SM90.
2026-10-19 20:16:46 +08:00
minke.yu ea5fc832c1 scheduler: CP-symmetric idle check for health-check admission
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.

Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).

(cherry picked from commit b91137ab98)
2026-10-19 10:47:57 +08:00
minke.yu 9894dc789c G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
  pool_transfers (comp prepare_prefetch + build_hicache_transfers +
  sidecars) so DeepSeek V4's logical-anchor KV goes through
  batch_exists_v2 side-pool existence instead of v1 batch_exists which
  always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
  backend is attached (not only decode-hicache mode; SWA models forbid
  decode radix+hicache so P nodes could never query L3); construct
  RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
  disagg-prefill result path (it bypasses batch_result_processor, so
  drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.

(cherry picked from commit e7dc175e94)
2026-10-19 10:47:07 +08:00
minke.yu 77e7555ae3 G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。

(cherry picked from commit 71c6d8bc55)
2026-10-19 10:46:23 +08:00
minke.yu 319ee3c056 G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
  记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
  set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费

B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
  + query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
  _tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
  v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit

(cherry picked from commit 4eabf5e228)
2026-10-19 10:46:23 +08:00
minke.yu fd86688428 PR #40219: DeepSeek V4.1 vision CP and single-owner image encoding
Source: https://github.com/sgl-project/sglang/pull/40219 (applied via GitHub .diff, 3way)
Adaptations for v0.5.21 base:
- attn_dp_enabled_of(cfg) -> cfg.enable_dp_attention (#41818 not in base)
- dropped _validate_joint_schema_request call (#42721 Clef, not in base)
- moved Backend import before the TRT-LLM prefill-graph check
2026-10-19 10:45:10 +08:00
Vladislav Nosivskoy da343ba5d1 [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 (#42604)
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2ff3e6dce1)
2026-10-19 10:38:40 +08:00
minke.yu 9ea02a7201 PR #40421: reuse shared compressed KV dequantization in PD prefill
Source: https://github.com/sgl-project/sglang/pull/40421 (applied via GitHub .diff, 3way)
2026-10-19 10:37:44 +08:00
minke.yu 6c1c54b514 fixup: import is_gfx95_supported for #41294 and #40860 ROCm branches
dsv41_sparse.py and graph_variants.py referenced is_gfx95_supported without
importing it (latent NameError on the ROCm path).
2026-10-19 10:37:08 +08:00
minke.yu 51624199b4 PR #41294: use the fused low-ratio compressor on Hopper (sm90 gate)
Source: https://github.com/sgl-project/sglang/pull/41294 (applied via GitHub .diff, 3way)
Skipped: test_dsv41_fused_compress.py (file not in v0.5.21 base).
2026-10-19 10:35:29 +08:00
minke.yu 31b3758c99 PR #42365: fuse Hopper prefill indexer scoring into one Triton kernel
Source: https://github.com/sgl-project/sglang/pull/42365 (applied via GitHub .diff, 3way)
Conflicts resolved: unioned env flags with #40945 SM90 flags; merged fused
scoring loop with #40860 candidate_blocks path (candidate > fused > torch priority).
2026-10-19 10:33:13 +08:00
minke.yu 740372bef0 PR #40945: SM90 static verification indexing (grouped/length-aware indexer)
Source: https://github.com/sgl-project/sglang/pull/40945 (applied via GitHub .diff, 3way)
Adaptation: PR base predates #41659 which removed the Selection struct;
ported sm90_decode.py and call sites to the inputs.out_page_indices interface
(raw_indices=None on decode paths, reset via inputs.reset_outputs()).
2026-10-19 10:30:41 +08:00
minke.yu 19fe20baec PR #40860: SM90 candidate indexers (compact candidate dispatch)
Source: https://github.com/sgl-project/sglang/pull/40860 (applied via GitHub .diff, 3way)
Conflicts resolved: kept #41658 padded-return trim in fp4_index_logits_paged,
took PR side for candidate_blocks branches, SM90 graph gate, deep_select test param.
Skipped: test_dsv41_candidate_graph_variants.py (file not in v0.5.21 base).
2026-10-19 10:22:09 +08:00
yuyu5333 534842ac08 [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. (#41561)
Co-authored-by: Claude Code <noreply@anthropic.com>
(cherry picked from commit b50fdeb0ef)
2026-10-19 10:16:18 +08:00
Yuwei An 4dd95a8817 [DSV4.1] Reduce eager Engram extend preparation (#42159)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d434ea444d)
2026-10-19 10:16:08 +08:00
Yuwei An 656e00dcd3 [DSv4.1] mxfp8 dispatch cache (#42156)
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
(cherry picked from commit 2d38934493)
2026-10-19 10:15:46 +08:00
DarkSharpness a4b1363bf4 [DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs (#41659)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 520d6d0229)
2026-10-19 10:15:46 +08:00
DarkSharpness ad2acaa2df [DSv4.1] Fold q_rope_store into fused_q_norm_rope (#41657)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 379ec90fb2)
2026-10-19 10:15:45 +08:00
Yuwei An ecaaa37cb0 [Fix][DSV4.1] SWA page size with bounded replay (#42128)
(cherry picked from commit ef867fa40d)
2026-10-19 10:14:30 +08:00
DarkSharpness 1b70fc192c [DSv4.1] Trim prefill model-entry overhead (#41704)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 2f6e0ec45d)
2026-10-19 10:14:29 +08:00
DarkSharpness 3ebbb2d536 [DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode (#41660)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit 5916999afb)
2026-10-19 10:14:29 +08:00
DarkSharpness 5d52cccbee [DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices (#41658)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BBuf <1182563586@qq.com>
(cherry picked from commit d75d5b33f7)
2026-10-19 10:14:28 +08:00
minke.yu 74d03143e2 PR #41098: H20 block-FP8 tuned GEMM configs
Source: https://github.com/sgl-project/sglang/pull/41098 (applied via GitHub .diff, 3way)
2026-10-19 10:13:59 +08:00
Xiaoyu Zhang 2b192aade6 [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (#41251)
(cherry picked from commit 391e665dab)
2026-10-19 10:13:35 +08:00
minke.yu 9247c97919 PR #40919: fix dspark folded result buffer reuse race
Source: https://github.com/sgl-project/sglang/pull/40919 (applied via GitHub .diff, 3way)
2026-10-19 10:12:23 +08:00
minke.yu 5e7190b87a PR #40758: escape literal DSV4.1 image placeholder in user text
Source: https://github.com/sgl-project/sglang/pull/40758 (applied via GitHub .diff, 3way)
2026-10-19 10:12:16 +08:00
e00930c548 [Cherry-pick to release/v0.5.21] [Test] Demote PD test RDMA openability check to a warning (#41681) (#41802)
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 19:44:12 -07:00
Kangyan-ZhouandBaizhou Zhang d7ff74b984 [Cherry-pick to release/v0.5.21] [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL (#41777) (#41788)
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2026-09-29 17:30:38 -07:00
Jiacong Fang bd78095030 [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B (#41059) 2026-09-29 00:38:58 -07:00
MickandMick Qian dc1bd46802 [diffusion] feat: support bounded exact conditioning cache across native models (#40470)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-29 15:27:22 +08:00
eeecho c7be3e935b [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts (#34528) 2026-09-28 23:08:44 -07:00
Anupa SajikumarandMa Mingfei 2ff52e3ceb [Intel][XPU]Enable chunked prefill scnearios for XPU with UT (#33804)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:28:25 +08:00
9ae43b0eee [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path (#40828)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com>
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-29 13:16:03 +08:00
Cheng Wan 5781206833 [PD] Give FakeKVReceiver ensure_abort_notified (#41618) 2026-09-28 21:59:17 -07:00
Cheng Wan 6dad152490 [Refactor] Document layer boundary contracts and integration (#41557) 2026-09-28 20:18:27 -07:00
Cheng Wan de9f850b36 [Refactor] Rename the module to layer_boundary (#41555) 2026-09-28 20:17:50 -07:00
Cheng Wan 701213c867 [Refactor] Retire the layer facade and simplify boundary internals (#41554) 2026-09-28 20:17:33 -07:00
Cheng Wan 9b89b96230 [Refactor] Migrate specialized decoder and overlap boundaries (#41553) 2026-09-28 20:17:17 -07:00
Cheng Wan 49b5e76e45 [Refactor] Construct independent decoder stage boundaries (#41552) 2026-09-28 20:17:00 -07:00
Cheng Wan acffb0d20d [Refactor] Select reduction fusion at the consumer (#41551) 2026-09-28 20:16:44 -07:00
Cheng Wan cb1fa3c399 [Refactor] Capture auxiliary states at residual reads (#41550) 2026-09-28 20:16:26 -07:00
Cheng Wan 42f58b4da7 [Refactor] Carry residual state across stage boundaries (#41549) 2026-09-28 20:15:03 -07:00
Cheng Wan 6f72cc3b64 [Refactor] Centralize decoder output access (#41548) 2026-09-28 20:14:04 -07:00
Cheng Wan 0a8ca85565 [Refactor] Group communicator fusion and CP adapters (#41547) 2026-09-28 20:10:50 -07:00
Liangsheng Yin d9074f7943 [Test] Fail fast when PD test RDMA devices are not openable by ibverbs (#41600) 2026-09-28 19:34:58 -07:00
James 6e1df69623 [npu]support NPU 910C L2 memcache offload (#41527) 2026-09-29 10:30:01 +08:00
77091cea68 [Radix Cache] Sync Rust TreeCore and make it the default (#39627)
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: alphabetc1 <alphabetc1@users.noreply.github.com>
2026-09-29 10:18:57 +08:00