minke.yu
0afcb334a4
realstore 回退 STORAGE_BATCH_SIZE=512:TCP 传输下大批次倒退(mc8 668k→320k),dummy shm 才受益;真实 store 保持默认 128
2026-10-10 18:16:56 +08:00
minke.yu
33e6381b7d
tier dummy compose 对齐 b300 实况:wait_complete + HC_TIMING + STORAGE_BATCH_SIZE=512
2026-10-10 18:06:48 +08:00
minke.yu
9c4ff32562
tier compose:STORAGE_BATCH_SIZE=512(摊薄每批 ~60ms 固定开销,dummy 实测 mc8 +26%)
2026-10-10 18:05:16 +08:00
minke.yu
ac11e9162d
tier realstore: extra_hosts 别名让 rank 后缀主机名可解析
...
mooncake P2P 模式下段名/te_endpoint 内嵌 local_hostname 且需 DNS 可解析;
rank 后缀名(localhost-d0t0p0c0 等)必须配 /etc/hosts 别名指向 127.0.0.1,
否则跨段 openSegment 握手失败(TRANSFER_FAIL 全灭)。
验证:mc8 402k→668k tok/s,读路径约 40% 走 memcpy 60% 仍 TCP。
2026-10-10 17:14:18 +08:00
minke.yu
8d9bce30a7
tier realstore: 开 HC_TIMING 打点 + prefetch policy 改 wait_complete
...
best_effort 下低并发请求 compute 赢比赛、L3 命中不生效(mc1/2/4 实为重算);
wait_complete 后真实 L3 命中:mc1 TTFT 2989→661ms,mc8 聚合 203k→402k tok/s。
2026-10-10 15:54:11 +08:00
minke.yu
f4ef648b16
tier L3 A/B:dummy→真实 mooncake store(线上同构 master+P2PHANDSHAKE+tcp)+ 零预热 mc 扫描脚本
...
dummy 后端实测单请求 L3 命中 2.96s/54k tok/s(零预热修正后),聚合 ~200k;
CPU 在 scheduler native 线程、store 守护进程与 Python 工作线程均空闲,
疑似 dummy 后端 batch_get 内部 per-key 串行。切真实 store 复核以区分
sglang L3 路径问题 vs dummy 调试后端工件。注意 bench 必须 --warmup-requests 0,
默认 warmup 会把 prompt#0 种进 L1/L2 污染 L3 测量。
2026-10-10 13:59:31 +08:00
minke.yu
6469cf4197
deploy: tier bench scripts (cp2 L1/L2/L3 matrix + R7 async aggregation)
2026-10-10 10:50:24 +08:00
minke.yu
3f8751c6ed
新增 tier-cp2 单机 compose:cp2+dspark+L2 6.4M+L3 shm dummy,测 计算/L1/L2/L3 四档 prefill 极限与异步聚合
2026-10-10 09:51:09 +08:00
minke.yu
e4f1310d6f
l3dummy: hicache-ratio 1.0→4.0(L2=3.2M token/P)——#38348 bench 显示收益被 L2 容量掩盖(host_capacity 拒绝 0.75M~2.97M token/P),放大 L2 验证热轮命中不被容量拒、L3 hit 嫁接后在 L2 留存
2026-10-10 09:14:07 +08:00
minke.yu
2b2ede8211
l3dummy 定型为 shm+best_effort(唯一正收益组合 +26.4%);wait_complete 因 discard 白等 -17.7% 弃用
2026-10-10 00:09:02 +08:00
minke.yu
b4dbcebbec
mooncake standalone(dummy) 模式打通 dsv41 unified cache:放宽 standalone 检查对 LogicalHostPool 锚点的误拦(实体 buffer 由旁池运行时 register_buffer);新增 l3dummy compose(单 store 服务 64GB shm 段 + fake-cargo 跳过 rustup 下载)。实测 24x160k 热/冷 +26.4%(vs TCP best_effort -4.4%)
build-sglang-image / build (push) Failing after 1h10m35s
2026-10-10 00:06:32 +08:00
minke.yu
30ba5818e7
l3test: prefetch 策略 wait_complete→best_effort(实测等待阻塞净亏 7-17%,best_effort 持平 -4.4%)
build-sglang-image / build (push) Failing after 1s
2026-10-09 20:30:46 +08:00
minke.yu
c9bfb24bcc
l3 模式:默认 48×160k + 冷热轮间 120s 备份排空
2026-10-09 19:31:45 +08:00
minke.yu
3d109d50be
fix: l3test L2 改 ratio 1.0(0.05 会噎死 L3 写入路径)
2026-10-09 19:30:53 +08:00
minke.yu
7d67d57804
fix: l3test 变体改用 hicache-ratio 0.05(V4 不支持 hicache-size)
2026-10-09 17:55:02 +08:00
minke.yu
491989284f
deploy: 3p1d-l3test 变体(P 缩 L1/L2 强制 L3 命中)
2026-10-09 17:26:04 +08:00
minke.yu
53ea8fb2a7
report: 增加 input tps 列
2026-10-09 17:02:23 +08:00
minke.yu
efdbfd2cc3
fix: awk sprintf 括号+TPOT/TTFT 取最后一列
2026-10-09 17:01:35 +08:00
minke.yu
8342b89e85
fix: awk sprintf 括号
2026-10-09 17:01:05 +08:00
minke.yu
c92938d718
deploy: bench_suite_v2 (warmup/prefill冷热双轮/decode阶梯80%cg/L3语料溢出) + 单独P/D基线compose
2026-10-09 16:50:23 +08:00
minke.yu
7578cdd218
revert: #42035 continuous prefill input polling (CP rank collective mismatch deadlock)
...
build-sglang-image / build (push) Failing after 39m1s
compose 3p1d regression on 1f6ca2cf6 : all 3 P nodes wedged with split-brain
CP ranks (one blocked in _sync_hicache_ready_counts all_reduce, partner in
request broadcast). Root cause: #42035 wraps get_next_disagg_prefill_batch_to_run
(which contains the hicache attn_cp_group all_reduce) in rank-local guards
(len(result_queue)<2, _deferred_input_requests, skip_batch_creation); any
cross-rank divergence in batch-None decisions misaligns collective pairing
permanently. Continuous polling requires attn_cp_size==1 anyway, so cp2
deployments get zero benefit. Reverts 93d610c1fd + test adaptations
6ba34507f1/bde1fcbf51/1f6ca2cf68 + deploy bump 236686454c .
Keeps #42194 (9ef65b51c2 ) and checkpoint_disagg_prefill shim (c4f3e3be33 ).
Evidence: D:\B300\investigations\pd-deadlock-20261009.md
2026-10-09 16:39:06 +08:00
minke.yu
21f7ce37dd
build: 镜像内编译 sgl-model-gateway(Rust gateway,G9 前置)
...
build-sglang-image / build (push) Failing after 31m52s
三个 Dockerfile(b300 全量/fast/gitea)在 COPY 源码后加 cargo 层:
cargo build --release -p sgl-model-gateway -p sgl-model-gateway-python,
RUSTUP_TOOLCHAIN=stable 绕过 rust-toolchain.toml 1.90 钉版(基底自带 1.98.1),
BuildKit cache mount 持久化 registry/target(冷 ~15min,增量秒~分钟级),
python 扩展产物重命名 sglang_router_rs.abi3.so 覆盖 python 包内同文件。
2026-10-09 b300 实测:crates.io 直连无需镜像,冷编译成功。
2026-10-09 16:07:36 +08:00
minke.yu
236686454c
deploy: bump bench compose templates to 1f6ca2cf6 (#42035+#42194)
2026-10-09 15:06:41 +08:00
minke.yu
4d0efcc921
deploy: bring k8s 3p1d RBG cw manifest into git; bump to 1068926cd (merged upstream batch)
...
Source of truth moves from D:\B300\build-kmctl (untracked) to
deploy/b300-ds41/ds41-3p1d-rbg-cw.yaml. Image carries A-batch (7) + B-batch (4)
upstream cherry-picks + is_hip_runtime import fix; bench-validated on compose
(3p1d 8-conc 160k 98903 tok/s, +50% vs 0928 baseline).
2026-10-08 17:41:06 +08:00
minke.yu
f71fafa684
deploy: bump to 1068926cd (is_hip_runtime import fix)
2026-10-08 16:39:04 +08:00
minke.yu
1068926cd6
fix: import is_hip_runtime in dsv4 low_ratio_compress (cherry-pick #41660 lost it)
...
build-sglang-image / build (push) Successful in 32m18s
Upstream's file gets is_hip_runtime from sglang.kernels.jit.utils via an
intermediate refactor we don't have; our cherry-pick applied cleanly but
the resulting file references is_hip_runtime() without importing it,
crashing every prefill (NameError in c2_prefill_norm_rope_store).
Also unify Dockerfile.fast with the v3.1 full path (check_deps + --no-deps
delta installs) so fast builds after a kernel-wheel bump don't silently
revert to the base image's older kernels.
2026-10-08 16:31:34 +08:00
minke.yu
48fb49baef
deploy: bump cpdspark-nr/b4-dp/3p1d-otlp to dsv41-pd-visioncp-2372cdc06-local-20261008-0808 (merged upstream fixes batch)
2026-10-08 16:21:30 +08:00
minke.yu
2372cdc06e
build: light delta also --no-deps — any dep-graph walk over torch re-downloads pinned nccl==2.29.7 over base's 2.30.7
2026-10-08 16:07:50 +08:00
minke.yu
7b815cc98b
build: COPY check_deps.py from sglang/ tree (context .dockerignore excludes build/)
2026-10-08 16:01:12 +08:00
minke.yu
d06c5dde42
build: v3.1 three-class dep delta — kernel wheels (sglang-kernel/flashinfer/sgl-deep-gemm) install via pip --no-deps when pins move
...
Base ships sglang-kernel 0.4.6.post1 but pyproject pins ==0.4.7; kernel
wheels must track the sglang source, so they are an allowed delta (~370MB
wheel), unlike the torch/nvidia family which must always reuse the base.
Also confirmed 2026-09-24 full build silently downgraded NCCL 2.30.7->2.29.7;
v3 keeps base NCCL untouched.
2026-10-08 15:59:36 +08:00
minke.yu
8d0ee5e51e
build: v3 full path reuses base torch/nvidia via local delta check
...
build-sglang-image / build (push) Successful in 28m50s
Full-resolution pip walks installed torch's metadata and would re-download
the pinned nvidia family (~6GB, throttled to 2h40m on 2026-09-23) and
downgrade the deliberately newer nvidia-nccl-cu13 2.30.7 to ==2.29.7.
v3 replaces pip full-resolution with check_deps.py: compare pyproject
requirements against base-installed versions locally, pip-install only the
lightweight delta via -r, and fail loudly if a heavy pin (torch*/nvidia-*/
sglang-kernel/flashinfer/cuda-*) becomes unsatisfied. Editable install keeps
--no-deps --no-build-isolation.
2026-10-08 15:54:58 +08:00
minke.yu
e403cede0a
deploy: D mr320/cg256/mf0.83 (KV pool 41.8M tok/worker); d1 HTTP 30001->30030 fix port collision with p1 scheduler loopback bind
2026-09-28 20:00:11 +08:00
minke.yu
735434fcde
deploy: k8s RBG 3p1d cw manifest; D cg512 + load-balance-method total_requests
2026-09-28 16:47:16 +08:00
minke.yu
e17fbc9a2d
deploy: 3p1d-otlp cs32k variant (chunked_prefill 32768 on P)
...
A/B vs dockerserve-3p1d-otlp.yml (chunk 16384), same image dsv41-pd-visioncp-db7d2cb7d-fix:
- bs1 160k prefill 61950 -> 66084 tok/s (+6.7%)
- bs16 4k prefill 47725 -> 70463 tok/s (+47.6%, fewer scheduler rounds)
- decode TPOT within dspark noise; trace: 160k sender 10->5 chunks, 3.16s->2.98s
Bench logs: bench_cs32k-20260927.log (round2) vs bench_otlptrace2-20260927.log
2026-09-27 14:31:33 +08:00
minke.yu
5507fff133
deploy: add 3p1d soak/prof compose + otlp tracing variant (jaeger badger 6h)
2026-09-27 12:10:00 +08:00
minke.yu
4f0ca95477
deploy: b300 build scripts with pip cache mount + no-build-isolation fast path
...
Base image already has torch 2.13, but pyproject build-system.requires
pulls it into pip's isolated build env on every build (~6GB via throttled
mirror, 47min). --no-build-isolation reuses the base env; build.sh picks
Dockerfile.fast (--no-deps) when python/pyproject.toml is unchanged.
2026-09-24 13:11:41 +08:00
minke.yu
67d8368a84
deploy: archive b300 ds41 compose files; skip CI for deploy/ and .gitea/ changes
2026-09-24 11:49:37 +08:00