minke.yu
4eabf5e228
G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
...
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费
B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
+ query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
_tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit
2026-10-09 18:48:27 +08:00
minke.yu
7578cdd218
revert: #42035 continuous prefill input polling (CP rank collective mismatch deadlock)
...
build-sglang-image / build (push) Failing after 39m1s
compose 3p1d regression on 1f6ca2cf6 : all 3 P nodes wedged with split-brain
CP ranks (one blocked in _sync_hicache_ready_counts all_reduce, partner in
request broadcast). Root cause: #42035 wraps get_next_disagg_prefill_batch_to_run
(which contains the hicache attn_cp_group all_reduce) in rank-local guards
(len(result_queue)<2, _deferred_input_requests, skip_batch_creation); any
cross-rank divergence in batch-None decisions misaligns collective pairing
permanently. Continuous polling requires attn_cp_size==1 anyway, so cp2
deployments get zero benefit. Reverts 93d610c1fd + test adaptations
6ba34507f1/bde1fcbf51/1f6ca2cf68 + deploy bump 236686454c .
Keeps #42194 (9ef65b51c2 ) and checkpoint_disagg_prefill shim (c4f3e3be33 ).
Evidence: D:\B300\investigations\pd-deadlock-20261009.md
2026-10-09 16:39:06 +08:00
minke.yu
c4f3e3be33
refactor: route disagg prefill checkpoints through checkpoint_disagg_prefill shim
...
Upstream #42035 's tests and call structure expect Scheduler.checkpoint_disagg_prefill
(introduced upstream with #41520 's checkpoint_kv_cache / insert_req cache API).
This branch still uses maybe_cache_unfinished_req (old cache_unfinished_req API),
so add a thin shim with the upstream name and route the three call sites through it.
No behavior change.
2026-10-09 11:22:42 +08:00
Liangsheng Yin
9ef65b51c2
[mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public ( #42194 )
...
build-sglang-image / build (push) Successful in 25m53s
(cherry picked from commit 07e1a4f082 )
2026-10-09 11:18:17 +08:00
metamergebot
93d610c1fd
[PD] Keep ingesting requests while a prefill forward result is pending ( #42035 )
...
build-sglang-image / build (push) Successful in 26m6s
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
(cherry picked from commit 6acf2f4736 )
2026-10-09 10:56:14 +08:00
minke.yu
1068926cd6
fix: import is_hip_runtime in dsv4 low_ratio_compress (cherry-pick #41660 lost it)
...
build-sglang-image / build (push) Successful in 32m18s
Upstream's file gets is_hip_runtime from sglang.kernels.jit.utils via an
intermediate refactor we don't have; our cherry-pick applied cleanly but
the resulting file references is_hip_runtime() without importing it,
crashing every prefill (NameError in c2_prefill_norm_rope_store).
Also unify Dockerfile.fast with the v3.1 full path (check_deps + --no-deps
delta installs) so fast builds after a kernel-wheel bump don't silently
revert to the base image's older kernels.
2026-10-08 16:31:34 +08:00
metamergebot
75758cb0a0
[HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails ( #42420 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com >
(cherry picked from commit 4ab720e655 )
2026-10-08 15:41:51 +08:00
metamergebot
60e80ac7d1
[HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert ( #42264 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
(cherry picked from commit 2bae12b9b3 )
2026-10-08 15:37:43 +08:00
Sasha Sidorov
4ef799aaa6
[PD] Wait for prefill completion before Mooncake early KV transfer ( #41395 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
(cherry picked from commit c683190b40 )
2026-10-08 15:36:21 +08:00
Aurick Qiao
1be762368b
[Disagg] Validate state strides before Mooncake transfers ( #41607 )
...
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com >
(cherry picked from commit 1a0250c5b6 )
2026-10-08 15:35:05 +08:00
DarkSharpness
f577b61b52
[DSv4.1] Trim prefill model-entry overhead ( #41704 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 2f6e0ec45d )
2026-10-08 15:32:48 +08:00
DarkSharpness
4792b61332
[DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode ( #41660 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit 5916999afb )
2026-10-08 15:32:48 +08:00
DarkSharpness
62f1ed099e
[DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices ( #41658 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
(cherry picked from commit d75d5b33f7 )
2026-10-08 15:32:47 +08:00
AMD-yanfeiwang
bfd9bb2e03
fix(cuda-graph): remove unnecessary CP batch alignment ( #40543 )
...
(cherry picked from commit 3dc7f4b816 )
2026-10-08 15:32:46 +08:00
metamergebot
6deac260bf
[PD] Make the prebuilt last-token H2D copy non-blocking under overlap ( #42113 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: bbus <bbus@users.noreply.github.com >
(cherry picked from commit 7dd93b01a5 )
2026-10-08 15:32:46 +08:00
Yuwei An
842ad7c4d4
[Fix][DSV4.1] SWA page size with bounded replay ( #42128 )
...
(cherry picked from commit ef867fa40d )
2026-10-08 15:32:45 +08:00
Mohammad Miadh Angkad
b605d2d370
[DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer ( #41987 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
(cherry picked from commit 42f2e4fcb7 )
2026-10-08 15:32:44 +08:00
minke.yu
b91137ab98
scheduler: CP-symmetric idle check for health-check admission
...
build-sglang-image / build (push) Successful in 27m52s
Health-check admit/skip used is_fully_idle(), which includes rank-local
hicache drain queues; ranks diverge right after activity, so one rank
dispatched the health-check generate while others piggyback-skipped,
deadlocking CP (hicache drain all_reduce vs CP request broadcast).
Seen on cp2/cp4 + hicache L3 after router health checks.
Recovered from b300 /data/ymk/sglang working copy (uncommitted WIP).
2026-09-24 12:01:57 +08:00
minke.yu
db7d2cb7db
fix: vision check uses get_parallel().pp_group (get_pp_group undefined in this tree)
build-sglang-image / build (push) Successful in 27m58s
2026-09-23 18:05:57 +08:00
Xinyuan Tong
6833498646
model: prune comments and redundant tests in dsv41 vision CP
2026-09-23 14:36:01 +08:00
Xinyuan Tong
b48e2cb1eb
model: TP-wide single-owner image encoding for DeepSeek V4.1
...
ViT and Aligner are replicated per TP rank, encoding each image eight
times with TP8 on both CP1 and CP8. Elect one owner per image and use
ordered full-span broadcasts with a six-phase agreement protocol.
Both CP1 and CP8 benefit while local cache hits preserve collective order.
2026-09-23 14:36:01 +08:00
Xinyuan Tong
bfeb7cd9b2
model: support DeepSeek V4.1 vision with interleave prefill CP
...
The CP runner bypassed the vision merge and used bare text embeddings.
Merge image features before sharding so request-global offsets stay valid.
Canonicalize model IDs separately to preserve scheduler hash IDs.
Keep unsupported combinations guarded and isolate embedding overrides
from multimodal prefills without starving queued FCFS requests.
2026-09-23 14:36:00 +08:00
Yuwei An
ddf5207630
[Fix] Handle chunked paged MQA metadata in DSV4.1 eager forwards ( #40637 )
build-sglang-image / build (push) Successful in 32m15s
2026-09-23 13:35:55 +08:00
minke.yu
b081dd3d23
Merge branch 'main' into dsv41-pd
2026-09-22 14:56:35 +08:00
Kevin Mi
8ac19cc19f
[AMD][Kimi-K3] Fix deferred KDA gate projection and update DCP cookbook ( #39066 )
PR Test (Arm64) / check-changes (push) Successful in 10s
PR Test (NPU) / set-image-config (push) Successful in 1s
PR Test (NPU) / Recommend tests from coverage (push) Skipped
PR Test (NPU) / check-changes (push) Successful in 12s
PR Test (sgl-router) / gate (push) Successful in 8s
PR Test (Xeon) / check-changes (push) Successful in 9s
PR Test (XPU) / check-changes (push) Successful in 16s
pr-test-arm64.yml / pr-gate (push) Successful in 3s
PR Test (Arm64) / pr-gate (push) Successful in 3s
pr-test-npu.yml / pr-gate (push) Successful in 2s
PR Test (NPU) / pr-gate (push) Successful in 2s
PR Test (sgl-router) / tier-1 — lint (push) Failing after 33s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 3s
PR Test (Xeon) / pr-gate (push) Successful in 3s
PR Test (sgl-router) / finish (push) Successful in 1s
Lint / lint (push) Failing after 2m51s
PR Test (NPU) / base-a-test-1-npu-a2 (push) Canceled after 0s
PR Test (NPU) / base-b-test-1-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-b-test-2-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-b-test-4-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-b-test-8-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-b-test-16-npu-a3 (push) Canceled after 0s
PR Test (NPU) / multimodal-gen-test-1-npu-a3 (0) (push) Canceled after 0s
PR Test (NPU) / multimodal-gen-test-1-npu-a3 (1) (push) Canceled after 0s
PR Test (NPU) / multimodal-gen-test-4-npu-a3 (0) (push) Canceled after 0s
PR Test (NPU) / multimodal-gen-test-4-npu-a3 (1) (push) Canceled after 0s
PR Test (NPU) / base-c-test-acc-2-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-c-test-acc-16-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-c-test-perf-2-npu-a3 (push) Canceled after 0s
PR Test (NPU) / base-c-test-perf-16-npu-a3 (push) Canceled after 0s
PR Test (NPU) / Analyze failure report (push) Canceled after 0s
PR Test (NPU) / setup-covstub (push) Canceled after 0s
PR Test (NPU) / pr-test-npu-finish (push) Canceled after 0s
pr-test-npu.yml / run (${{ fromJson(inputs.partitions).arr }}) (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
PR Test (Arm64) / build-test (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 0, 3, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 1, 3, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr3, 2, 3, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
2026-09-22 06:35:12 +00:00
Mohammad Miadh Angkad and Mohammad Angkad
4c81cd1b09
[KDA] Fix missing beta sigmoid in PTX prefill ( #40685 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-21 23:24:38 -07:00
Khoa Pham
bc22e1de9e
[DSpark] Fix draft CUDA graph stream explosion ( #40658 )
2026-09-21 23:05:45 -07:00
Guangda Liu and Guangda Liu
04c0913434
[HiSparse] Add MHA hisparse support for MiniMax M3 ( #31446 )
...
Co-authored-by: Guangda Liu <bingps@users.noreply.github.com >
2026-09-22 13:28:03 +08:00
095e45100b
[AMD] [GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950 ( #38545 )
...
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-21 22:24:29 -07:00
Piotr Mazurek
b01961e295
[LFM2-VL] Add DSpark speculative decoding ( #40651 )
2026-09-21 21:49:51 -07:00
YAMY
9b59fc5db5
[ModelOpt][PP] Keep BF16 shared experts out of the NVFP4 fusion so TP1 pipeline stages can load ( #40628 )
2026-09-21 21:45:58 -07:00
a9f02b0fa4
[NPU] Fix xgrammar apply_vocab_mask device dispatch to use torch.ops.npu ( #36120 )
...
Co-authored-by: Even Zhou <even.y.zhou@outlook.com >
Co-authored-by: sglang-npu-bot <sglangnpu@163.com >
2026-09-21 21:34:15 -07:00
877a293d6d
[Benchmark] Optionally clear HiCache storage between cases ( #40659 )
...
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com >
Co-authored-by: Pengchao Wang <wpc@fb.com >
2026-09-21 21:31:21 -07:00
e1daf68304
[AMD] [GLM-5.3-Flash Day 0] Honor fused and per-expert names in quark exclude ( #39317 )
...
Co-authored-by: Yikai Zhang <ykzhang12@gmail.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-21 21:26:42 -07:00
Khoa Pham and Qiaolin Yu
018b73c7a0
[PD] Pack draft KV head slices for DCP transfers ( #40500 )
...
Co-authored-by: Qiaolin Yu <liin1211@outlook.com >
2026-09-21 21:11:27 -07:00
b44e248682
[AMD] [GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950 ( #38546 )
...
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-21 21:07:37 -07:00
jthomson04
15ba54bd5d
perf(engine): avoid timed waits for Engine responses ( #39486 )
...
Signed-off-by: jthomson04 <jwillthomson19@gmail.com >
2026-09-21 20:42:03 -07:00
Jan Bernlöhr and Po-Han Huang
56fee88e23
fix(moe): support Llama4 NVFP4 router input weights on SM120 ( #35504 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-09-21 20:18:11 -07:00
90cf471723
[AMD] [GLM-5.3-Flash Day 0] Support non-2048 top-k widths in the DSA page-table transform ( #39340 )
...
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-21 19:50:17 -07:00
5f9c6b9eb0
[diffusion] fix: separate a use-scoped layerwise release from release_all ( #40590 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-22 10:38:44 +08:00
Mick and Mick Qian
a1b2b976fe
[diffusion] CI: restore public Qwen-Image 2.1 TP2 E2E coverage ( #40507 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-09-22 10:38:00 +08:00
Yuhan Zhou
15eba3b464
Feat: Add TensorCast storage as a new HiCache backend ( #27265 )
2026-09-22 10:19:04 +08:00
jacky.cheng
bc30fa1759
[AMD][Fix] AgentX HIP TPOT regression when SGLANG_SIMULATE_ACC_LEN is set ( #40598 )
2026-09-21 19:16:15 -07:00
Mohammad Miadh Angkad and mmangkad
e332e1b84e
[Fix] Don't write conv state from the fused KDA verify kernel ( #39524 )
...
Co-authored-by: mmangkad <mohammad.angkad@radixark.ai >
2026-09-21 18:33:03 -07:00
Dayananda V and Claude Opus 5
35eb7cf8d6
[Intel][XPU][KVCanary] Enable KV Canary on Intel XPU ( #33520 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-22 09:19:01 +08:00
Khoa Pham
c4d3770a68
[Kimi K3] Fix CUDA graph stream explosion ( #40640 )
2026-09-21 17:49:50 -07:00
jacky.cheng
31b577bb08
[AMD] Pad QSA MQA decode Q-heads to 16 for ROCm MFMA ( #38875 )
2026-09-21 17:43:46 -07:00
042b6a488f
[AMD] [GLM-5.3-Flash Day 0] Enable zero-RoPE MHA prefill on ROCm ( #39338 )
...
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
2026-09-21 17:38:07 -07:00
Cheng Wan
582389cec5
[Fix] Keep diffusion encoder TP context bindings consistent ( #40646 )
2026-09-21 17:23:15 -07:00
Tao Li and Xiaoyu Zhang
c53cc8e1eb
[NPU][BugFix] Avoid M-RoPE recompilation for variable sequence lengths ( #40371 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-09-22 08:19:25 +08:00