minke.yu
f9056f0fcd
fix(dsv4 fp4_indexer): 128B-align SM90 paged score rows for deep_select
...
fp4_index_logits_paged padded rows to 8 floats (32B) and
fp4_index_logits_candidates did not pad at all, but the tvm-ffi
deep_select topk kernel enforces kInputAlignmentBytes=128 on the input
row stride. bs>1 decode/CUDA-graph capture died with
'Tensor stride for dimension 0 is not aligned to 128 bytes'.
Candidates kernel now takes a separate OUT_STRIDE constexpr.
2026-10-19 20:55:01 +08:00
minke.yu
2aae0cb4ee
fix(deep_gemm_wrapper): import DEEPGEMM_NEED_TMA_ALIGNED_SCALES/DEEPGEMM_SCALE_UE8M0 in compile_utils
...
Cherry-pick da343ba5d1 (#42604 ) lost the configurer import hunk; runtime
NameError killed every scheduler right after weight load on SM90.
2026-10-19 20:16:46 +08:00
minke.yu
ea5fc832c1
scheduler: CP-symmetric idle check for health-check admission
...
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.
Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).
(cherry picked from commit b91137ab98 )
2026-10-19 10:47:57 +08:00
minke.yu
9894dc789c
G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
...
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
pool_transfers (comp prepare_prefetch + build_hicache_transfers +
sidecars) so DeepSeek V4's logical-anchor KV goes through
batch_exists_v2 side-pool existence instead of v1 batch_exists which
always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
backend is attached (not only decode-hicache mode; SWA models forbid
decode radix+hicache so P nodes could never query L3); construct
RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
disagg-prefill result path (it bypasses batch_result_processor, so
drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.
(cherry picked from commit e7dc175e94 )
2026-10-19 10:47:07 +08:00
minke.yu
77e7555ae3
G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
...
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。
(cherry picked from commit 71c6d8bc55 )
2026-10-19 10:46:23 +08:00
minke.yu
319ee3c056
G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
...
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费
B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
+ query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
_tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit
(cherry picked from commit 4eabf5e228 )
2026-10-19 10:46:23 +08:00
minke.yu
fd86688428
PR #40219 : DeepSeek V4.1 vision CP and single-owner image encoding
...
Source: https://github.com/sgl-project/sglang/pull/40219 (applied via GitHub .diff, 3way)
Adaptations for v0.5.21 base:
- attn_dp_enabled_of(cfg) -> cfg.enable_dp_attention (#41818 not in base)
- dropped _validate_joint_schema_request call (#42721 Clef, not in base)
- moved Backend import before the TRT-LLM prefill-graph check
2026-10-19 10:45:10 +08:00
Vladislav Nosivskoy
da343ba5d1
[SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 ( #42604 )
...
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 2ff3e6dce1 )
2026-10-19 10:38:40 +08:00
minke.yu
9ea02a7201
PR #40421 : reuse shared compressed KV dequantization in PD prefill
...
Source: https://github.com/sgl-project/sglang/pull/40421 (applied via GitHub .diff, 3way)
2026-10-19 10:37:44 +08:00
minke.yu
6c1c54b514
fixup: import is_gfx95_supported for #41294 and #40860 ROCm branches
...
dsv41_sparse.py and graph_variants.py referenced is_gfx95_supported without
importing it (latent NameError on the ROCm path).
2026-10-19 10:37:08 +08:00
minke.yu
51624199b4
PR #41294 : use the fused low-ratio compressor on Hopper (sm90 gate)
...
Source: https://github.com/sgl-project/sglang/pull/41294 (applied via GitHub .diff, 3way)
Skipped: test_dsv41_fused_compress.py (file not in v0.5.21 base).
2026-10-19 10:35:29 +08:00
minke.yu
31b3758c99
PR #42365 : fuse Hopper prefill indexer scoring into one Triton kernel
...
Source: https://github.com/sgl-project/sglang/pull/42365 (applied via GitHub .diff, 3way)
Conflicts resolved: unioned env flags with #40945 SM90 flags; merged fused
scoring loop with #40860 candidate_blocks path (candidate > fused > torch priority).
2026-10-19 10:33:13 +08:00
minke.yu
740372bef0
PR #40945 : SM90 static verification indexing (grouped/length-aware indexer)
...
Source: https://github.com/sgl-project/sglang/pull/40945 (applied via GitHub .diff, 3way)
Adaptation: PR base predates #41659 which removed the Selection struct;
ported sm90_decode.py and call sites to the inputs.out_page_indices interface
(raw_indices=None on decode paths, reset via inputs.reset_outputs()).
2026-10-19 10:30:41 +08:00
minke.yu
19fe20baec
PR #40860 : SM90 candidate indexers (compact candidate dispatch)
...
Source: https://github.com/sgl-project/sglang/pull/40860 (applied via GitHub .diff, 3way)
Conflicts resolved: kept #41658 padded-return trim in fp4_index_logits_paged,
took PR side for candidate_blocks branches, SM90 graph gate, deep_select test param.
Skipped: test_dsv41_candidate_graph_variants.py (file not in v0.5.21 base).
2026-10-19 10:22:09 +08:00
yuyu5333
534842ac08
[DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. ( #41561 )
...
Co-authored-by: Claude Code <noreply@anthropic.com >
(cherry picked from commit b50fdeb0ef )
2026-10-19 10:16:18 +08:00
Yuwei An
4dd95a8817
[DSV4.1] Reduce eager Engram extend preparation ( #42159 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
(cherry picked from commit d434ea444d )
2026-10-19 10:16:08 +08:00
Yuwei An
656e00dcd3
[DSv4.1] mxfp8 dispatch cache ( #42156 )
...
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local >
(cherry picked from commit 2d38934493 )
2026-10-19 10:15:46 +08:00
DarkSharpness
a4b1363bf4
[DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs ( #41659 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 520d6d0229 )
2026-10-19 10:15:46 +08:00
DarkSharpness
ad2acaa2df
[DSv4.1] Fold q_rope_store into fused_q_norm_rope ( #41657 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
(cherry picked from commit 379ec90fb2 )
2026-10-19 10:15:45 +08:00
Yuwei An
ecaaa37cb0
[Fix][DSV4.1] SWA page size with bounded replay ( #42128 )
...
(cherry picked from commit ef867fa40d )
2026-10-19 10:14:30 +08:00
DarkSharpness
1b70fc192c
[DSv4.1] Trim prefill model-entry overhead ( #41704 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 2f6e0ec45d )
2026-10-19 10:14:29 +08:00
DarkSharpness
3ebbb2d536
[DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode ( #41660 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 5916999afb )
2026-10-19 10:14:29 +08:00
DarkSharpness
5d52cccbee
[DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices ( #41658 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit d75d5b33f7 )
2026-10-19 10:14:28 +08:00
minke.yu
74d03143e2
PR #41098 : H20 block-FP8 tuned GEMM configs
...
Source: https://github.com/sgl-project/sglang/pull/41098 (applied via GitHub .diff, 3way)
2026-10-19 10:13:59 +08:00
Xiaoyu Zhang
2b192aade6
[Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection ( #41251 )
...
(cherry picked from commit 391e665dab )
2026-10-19 10:13:35 +08:00
minke.yu
9247c97919
PR #40919 : fix dspark folded result buffer reuse race
...
Source: https://github.com/sgl-project/sglang/pull/40919 (applied via GitHub .diff, 3way)
2026-10-19 10:12:23 +08:00
minke.yu
5e7190b87a
PR #40758 : escape literal DSV4.1 image placeholder in user text
...
Source: https://github.com/sgl-project/sglang/pull/40758 (applied via GitHub .diff, 3way)
2026-10-19 10:12:16 +08:00
e00930c548
[Cherry-pick to release/v0.5.21] [Test] Demote PD test RDMA openability check to a warning ( #41681 ) ( #41802 )
...
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-09-29 19:44:12 -07:00
Kangyan-Zhou and Baizhou Zhang
d7ff74b984
[Cherry-pick to release/v0.5.21] [Fix] Restore deferred layer dumps and pin SentencePiece for InternVL ( #41777 ) ( #41788 )
...
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com >
2026-09-29 17:30:38 -07:00
Jiacong Fang
bd78095030
[Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B ( #41059 )
2026-09-29 00:38:58 -07:00
Mick and Mick Qian
dc1bd46802
[diffusion] feat: support bounded exact conditioning cache across native models ( #40470 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-09-29 15:27:22 +08:00
eeecho
c7be3e935b
[SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts ( #34528 )
2026-09-28 23:08:44 -07:00
Anupa Sajikumar and Ma Mingfei
2ff52e3ceb
[Intel][XPU]Enable chunked prefill scnearios for XPU with UT ( #33804 )
...
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-09-29 13:28:25 +08:00
9ae43b0eee
[XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path ( #40828 )
...
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com >
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com >
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-09-29 13:16:03 +08:00
Cheng Wan
5781206833
[PD] Give FakeKVReceiver ensure_abort_notified ( #41618 )
2026-09-28 21:59:17 -07:00
Cheng Wan
6dad152490
[Refactor] Document layer boundary contracts and integration ( #41557 )
2026-09-28 20:18:27 -07:00
Cheng Wan
de9f850b36
[Refactor] Rename the module to layer_boundary ( #41555 )
2026-09-28 20:17:50 -07:00
Cheng Wan
701213c867
[Refactor] Retire the layer facade and simplify boundary internals ( #41554 )
2026-09-28 20:17:33 -07:00
Cheng Wan
9b89b96230
[Refactor] Migrate specialized decoder and overlap boundaries ( #41553 )
2026-09-28 20:17:17 -07:00
Cheng Wan
49b5e76e45
[Refactor] Construct independent decoder stage boundaries ( #41552 )
2026-09-28 20:17:00 -07:00
Cheng Wan
acffb0d20d
[Refactor] Select reduction fusion at the consumer ( #41551 )
2026-09-28 20:16:44 -07:00
Cheng Wan
cb1fa3c399
[Refactor] Capture auxiliary states at residual reads ( #41550 )
2026-09-28 20:16:26 -07:00
Cheng Wan
42f58b4da7
[Refactor] Carry residual state across stage boundaries ( #41549 )
2026-09-28 20:15:03 -07:00
Cheng Wan
6f72cc3b64
[Refactor] Centralize decoder output access ( #41548 )
2026-09-28 20:14:04 -07:00
Cheng Wan
0a8ca85565
[Refactor] Group communicator fusion and CP adapters ( #41547 )
2026-09-28 20:10:50 -07:00
Liangsheng Yin
d9074f7943
[Test] Fail fast when PD test RDMA devices are not openable by ibverbs ( #41600 )
2026-09-28 19:34:58 -07:00
James
6e1df69623
[npu]support NPU 910C L2 memcache offload ( #41527 )
2026-09-29 10:30:01 +08:00
77091cea68
[Radix Cache] Sync Rust TreeCore and make it the default ( #39627 )
...
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
Co-authored-by: Zhangheng <hzh0425@apache.org >
Co-authored-by: alphabetc1 <alphabetc1@users.noreply.github.com >
2026-09-29 10:18:57 +08:00
WenhaoZhang
49d26ff0c2
[diffusion] model: support flux 3 action robot policies ( #41066 )
2026-09-29 10:17:19 +08:00
79cafec013
[Model] Add IQuest Q1 support and MTP draft ( #41590 )
...
Co-authored-by: zelong huang <yzhu@ubiquant.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: zelong518 <zelonghuang05@gmail.com >
Co-authored-by: chenwenxiaolive <cocoshirolive@gmail.com >
2026-09-28 17:21:31 -07:00