Commit Graph
18788 Commits
Author SHA1 Message Date
minke.yu ba3318ffbd gateway g9i: fix shadow compile errors — lookup returns owned snapshot (borrow across loop mutation), bind temp in test 2026-10-13 15:14:05 +08:00
minke.yu 8279d3d6df gateway g9i: v2 P1 shadow mode — prefix-hash chain, bookkeeping table, shadow classifier + queue model
Observability only, zero routing influence, gated by --g9i-shadow-enabled
(requires --g9i-enabled). Design: experiments/g9i/g9i-v2-design-20261013.md.

- g9i_prefix_chain.rs: rolling xxh3 chain over chat messages (seed folds
  model/template-kwargs/tools/system; tool_calls/tool_call_id/name hashed;
  byte-level image semantics; unclassifiable flag for unknown parts)
- g9i_shadow.rs: PrefixTable (LRU+TTL, deepest-hit lookup, exact tail
  anchor + placeholder middle boundaries), ShadowClassifier (hot/short/
  cold_image/cold_large/cold_unknown via byte heuristic), QueueModel
  (per-(node,class) outstanding + completion-driven drain EWMA, est_TTFT),
  usage/cached_tokens_details injection+strip pairing
- pd_router.rs: request-side shadow classify + agreement counters (shadow
  vs v1 probe verdict), response-side usage harvest on stream/non-stream,
  queue admit/complete/release conservation incl. abort paths
- main.rs: --g9i-shadow-enabled/-table-max-entries/-table-ttl-secs

Not yet compile-verified (no local toolchain); b300 container check next.
2026-10-13 15:05:56 +08:00
minke.yu ef4cccb6a0 gateway g9i: allow flow-control-only mode (no fast-P URL) — B case flat pool 2026-10-13 01:31:57 +08:00
minke.yu 98b950366d mooncake PD transfer: mix bootstrap_room into lane shard hash (fixes 1/16 lane collapse with single decode instance) 2026-10-12 23:41:23 +08:00
minke.yu 87ab6c0155 gateway g9i: probe fast-P only + L3-inclusive classification + exact-body probe cache (A案; removes accidentally reintroduced home_hit) 2026-10-12 13:03:54 +00:00
minke.yu 0308920bdb gateway g9i: retry probe once on transport errors (stale keep-alive pool)
Observed twice on 2026-10-12: reqwest reused server-closed pooled connections,
all-candidate probes failed with connect-level errors, fail-closed 429 drowned
hit-class traffic. Retry once on is_connect/is_request (ms-scale reconnect);
timeouts stay non-retried (wedged engine + 6s worst-case TTFT budget).
2026-10-12 12:02:49 +00:00
minke.yu 57db2a4ae9 gateway g9i: fail-closed 429 on probe failure / fast-P snapshot loss (p2 incident)
User directive 2026-10-12: 先严格429. When p2 hung, the fail-open fallback dumped
all classified traffic into the regular pool and drowned it (7.5M pending tokens).
All probe-failure paths now reject: prefill_probe_failed (429) when no candidate
can answer /query_cache_hit; fast_p_no_snapshot (429) when snapshots go stale
between probe and pick, with boot grace (had_any_success) mirroring the D gate.
2026-10-12 10:16:09 +00:00
minke.yu 7fd7fd58a5 gateway g9i: hit class always routes to fast-P pool (命中都在 fast-P 发生)
User directive 2026-10-12: read-your-own-node home routing sent hit-class
requests back onto the regular-P long-prefill queue (150s p50 observed),
defeating the fast path. Hit class now always goes to the fast-P pool
(least-wait aggregate); fast-P resolves the prefix from its own L2 or the
shared L3 (wait_complete), recompute-and-graft otherwise — self-healing
land-and-stay. Home override removed; fast-P overload is handled only by
the fast_p_queue 429 gate. New decision counter routed_fast_p_from_regular
distinguishes L3-fetch/recompute hits from local-L2 hits (observability).
2026-10-12 02:20:12 +00:00
minke.yuandKimi 71a72b8fea gateway g9i: keep large prefills off fast-P even without least-wait
regular_plan returned Default when regular_pool_least_wait=false, letting
the router policy pick across ALL prefill workers including the fast-P
node — ~1/3 of large prefills landed on fast-P under random policy
(observed 2026-10-12), polluting its small-prefill queue and L2. With a
fast-P configured, 'regular' now explicitly restricts to the regular
pool via the least-wait aggregate pick. Default is kept only when the
fast pool is empty (then it equals the regular pool anyway).

Co-authored-by: Kimi
2026-10-12 08:58:03 +08:00
minke.yuandKimi 6c9518fcd8 gateway g9i: move ProbeHit to module scope (fix compile)
Co-authored-by: Kimi
2026-10-12 08:20:30 +08:00
minke.yuandKimi 3a4ab451ac gateway g9i: classify on local (L1+L2) hits only, probe the whole pool
Follow-up to the home-hit routing fix. Two flaws remained:

1. The engine's /query_cache_hit new_tokens counts shared-L3 hits. Mooncake
   pages are metadata-visible (Exist) before the put finishes, so fast-P
   claimed long prefixes it could not actually serve: the gateway classified
   'small prefill, fast-P has it' while the engine prefetch found nothing
   and recomputed 76k cold — and the probe loop broke at fast-P before ever
   asking the home regular-P, which held an immediate L2 copy.

2. The probe loop stopped at the first answering node, so 'which node holds
   the prefix' was only ever 'fast-P or whoever got asked first'.

Now every fresh candidate is probed (fast pool first, early-out only on a
fully-local prompt), and the routing decision uses local_new = total - l1 -
l2. Ties keep probe order so fresh short prompts with no hits still land on
fast-P.

Co-authored-by: Kimi
2026-10-12 08:17:52 +08:00
minke.yuandKimi 5d38fc96a9 gateway g9i: route small prefills to the node that actually holds the prefix
classify_prefill remembered only new_tokens from the /query_cache_hit probe
and sent every small-prefill request to the fast-P pool. A multi-turn session
whose home P is a regular node therefore landed on fast-P with a local miss:
either a full prefix recompute or a wait for the cross-node L3 backup to
become visible (seconds for long prefixes; observed 10-20s for 76k).

Now the probe records which worker answered. When the prefix lives on a
regular-P the request is Override-routed home (immediate L2 hit), subject to
the same regular_p_queue est-wait 429 gate. Fast-P keeps serving sessions
whose prefix it already holds (its own L2) and brand-new short prefills.

Co-authored-by: Kimi
2026-10-12 08:01:10 +08:00
minke.yu 261ebbc55b G9i: regular-P queue admission gate (est wait > 10s -> 429 regular_p_queue) 2026-10-12 00:25:52 +08:00
minke.yu 53fb9be7ee G9i: TTFT p95/p99 statistics (observability only, no admission gate) 2026-10-11 22:31:30 +08:00
minke.yu e5be84f795 G9i: probe classification across pool candidates (fast-P then regular)
t15 stress through the gateway showed /query_cache_hit failing on the
sole fast-P under exactly the load where classification matters (p1
queue 62 / pending 4.7M tokens, 160 probe errors -> fallback). Probe
order is now fast-P least-wait first, regular pool as backup; first
successful answer classifies; unclassifiable (multimodal 400) still
short-circuits; total failure falls back to default routing.
2026-10-11 21:15:59 +08:00
minke.yu 39ceba8a3d G9i: router-side pool admission + fast-P routing (gateway)
Pool-aware admission control and small-prefill routing, entirely in
sgl-model-gateway (zero engine changes). Consumes G9 P0 engine surfaces:
/v1/loads (per-dp-rank snapshots, summed to per-node instance-level
views) and /query_cache_hit (new-token classification).

Decision tree per request, before PD dispatch and outside the retry
loop (own 429s are never retried):
  1. D-pool gate: sum of fresh num_running_reqs over decode pool >
     threshold -> 429 d_pool_saturated (fail-open on missing snapshots;
     engine max-running-requests is the hard backstop).
  2. Chat routes: probe least-wait fast-P /query_cache_hit; new_tokens
     < max_new_tokens (tentative 4096) -> fast-P pool.
  3. Pool selection: least estimated wait (queued_total_input_tokens /
     max(drain_rate, floor)) — doubles as the load balancer for the
     multi-fast-P / multi-D future; pool membership is config-only.
  4. Fast-P queue gate: least-wait estimate > threshold (tentative 5s)
     -> 429 fast_p_queue with x-estimated-wait-s.

CLI: --g9i-enabled/--g9i-fast-prefill-url/--g9i-max-new-tokens/
--g9i-fast-p-queue-reject-secs/--g9i-decode-max-running/... all numbers
tentative and centralized (user directive: fixed by measurement later).
Metrics: smg_g9i_decisions_total/rejects_total/no_fresh_snapshot_total,
per-node running gauge, probe latency histogram.
2026-10-11 19:14:40 +08:00
minke.yu b9d259441c deploy: l3test stack image -> e7dc175e9 (G9 P0 fixes baked in) 2026-10-11 17:47:24 +08:00
minke.yu e7dc175e94 G9 P0 fixes: v2 pool_transfers for storage hit query, disagg drain hook, query gate
build-sglang-image / build (push) Failing after 21m51s
- unified_radix_cache.query_storage_hit_length: build PREFETCH-phase
  pool_transfers (comp prepare_prefetch + build_hicache_transfers +
  sidecars) so DeepSeek V4's logical-anchor KV goes through
  batch_exists_v2 side-pool existence instead of v1 batch_exists which
  always misses (no physical KV objects are stored for V4).
- scheduler.handle_query_cache_hit: allow L3 query when hicache storage
  backend is attached (not only decode-hicache mode; SWA models forbid
  decode radix+hicache so P nodes could never query L3); construct
  RadixKey from array('q') to match tree node key types.
- disaggregation/prefill: record_prefill_drain on final chunk in the
  disagg-prefill result path (it bypasses batch_result_processor, so
  drain_rate_tokens_per_s stayed 0 on P nodes).
- deploy: 3p1d l3test compose against the real mooncake store stack.
2026-10-11 17:37:21 +08:00
minke.yu 6612af9e0e deploy: 归档 b300 实况——l3dummy 镜像升 baba18bc7d + D max-running-requests 512 2026-10-11 15:40:04 +08:00
minke.yu 71c6d8bc55 G9 P0 fix: /query_cache_hit 绕过 _tokenize_one_request 的 ReqState 依赖
_tokenize_one_request._create_tokenized_object 会读 rid_to_state[obj.rid]
(正常生成路径在 tokenize 前注册 ReqState),纯查询无 rid → KeyError: None。
改为:dsv41 chat encoding 直接渲染的 prompt_ids 直接用;否则对渲染后
text 调 _tokenize_texts(与真实路径同一分词入口)。tok8 环境实测 200。
2026-10-11 15:38:35 +08:00
minke.yu dbc661602e deploy: 默认开 SGLANG_HICACHE_PREFETCH_IO_THREADS=4(io_aux 4 lane,mc4/mc8 +23%)
- 11 个挂 L3 的 compose 模板 + k8s 3p1d RBG manifest 的 prefill-base 统一显式落 4
  (代码默认已是 4,显式化防漂移;同一实例所有 rank 必须一致——crc32(rid)%N 分发
  + 每 lane 独立 gloo 组,不一致会集合通信错位挂死)
- D 角色不挂 L3,无需加
- A/B 数据见 investigations/l3-hicache-deep-dive-20261010.md §8
2026-10-10 23:07:45 +08:00
minke.yu 054dcee896 tier compose:跨实例 P2P 的 extra_hosts 互配(A/B 互加对方 rank 主机名别名,否则副本跨端分配后 open segment DNS 失败) 2026-10-10 21:45:16 +08:00
minke.yu 73a25267ac tier realstore-x2 compose:第二实例(GPU 2,3 / 端口 30041 / localhost-b 前缀)做跨 P 实例 L3 读写验证 2026-10-10 21:28:21 +08:00
minke.yu bcad628d30 tier realstore compose: 显式 SGLANG_HICACHE_PREFETCH_IO_THREADS=4(io_aux lane 数,配合 baba18bc7d) 2026-10-10 20:05:54 +08:00
minke.yu baba18bc7d hicache L3 prefetch: io_aux 传输并行化(按 op 分 lane)
build-sglang-image / build (push) Failing after 48m34s
打点实测单条 prefetch_io_aux_func 线程串行传输是所有请求的瓶颈:
mc8 下每请求 xfer 排队 0->1.9s 线性增长(每请求 5 批 x ~68ms 固定开销,
介质无关)。本提交把传输层拆成 N 条并行 lane(默认 4,
SGLANG_HICACHE_PREFETCH_IO_THREADS 可调):

- dispatch_prefetch() 用 crc32(request_id) % N 分 lane(内置 hash 对 str
  每进程加盐,跨 rank 不一致会导致 gloo 集合通信错位,禁用)
- 同一 op 的所有 chunk 保持在同一 lane,保序 reduce 语义不变
- 每 lane 独立 io 队列 + io_aux 线程 + sync 队列 + sync 线程 +
  独立 gloo completion 组(集合通信按 lane 过滤后的分发顺序保序)
- query 阶段的 hit_reduce 仍在单 prefetch_thread,不动
- 生产点(hiradix/unified/hybrid pp-ticket)改调 dispatch_prefetch
- 模拟器与两个单测文件同步适配

注意:SGLANG_HICACHE_PREFETCH_IO_THREADS 必须在实例内所有
TP/CP/PP rank 保持一致,否则 lane 组上的集合通信会错位挂死。
2026-10-10 19:04:49 +08:00
minke.yu 0afcb334a4 realstore 回退 STORAGE_BATCH_SIZE=512:TCP 传输下大批次倒退(mc8 668k→320k),dummy shm 才受益;真实 store 保持默认 128 2026-10-10 18:16:56 +08:00
minke.yu 33e6381b7d tier dummy compose 对齐 b300 实况:wait_complete + HC_TIMING + STORAGE_BATCH_SIZE=512 2026-10-10 18:06:48 +08:00
minke.yu 9c4ff32562 tier compose:STORAGE_BATCH_SIZE=512(摊薄每批 ~60ms 固定开销,dummy 实测 mc8 +26%) 2026-10-10 18:05:16 +08:00
minke.yu 7d5759b2d9 hicache: STORAGE_BATCH_SIZE 可env化(SGLANG_HICACHE_STORAGE_BATCH_SIZE,默认128不变)
build-sglang-image / build (push) Failing after 18m40s
打点显示每批存储 IO 有 ~30-70ms 固定开销(与传输介质无关),
调大批次可摊薄。env 化便于不动代码做 128/512/1024 对照。
2026-10-10 17:55:14 +08:00
minke.yu ac11e9162d tier realstore: extra_hosts 别名让 rank 后缀主机名可解析
mooncake P2P 模式下段名/te_endpoint 内嵌 local_hostname 且需 DNS 可解析;
rank 后缀名(localhost-d0t0p0c0 等)必须配 /etc/hosts 别名指向 127.0.0.1,
否则跨段 openSegment 握手失败(TRANSFER_FAIL 全灭)。
验证:mc8 402k→668k tok/s,读路径约 40% 走 memcpy 60% 仍 TCP。
2026-10-10 17:14:18 +08:00
minke.yu 84fc632088 mooncake_store: local_hostname 加 rank 后缀,让 master 的 preferred-segment 分配落到本进程段
build-sglang-image / build (push) Failing after 1s
两个 rank 进程共用默认 localhost 时,master 按名字匹配无法区分两个段,
副本会落到对方进程段;mooncake 的 local memcpy 快路径要求同进程 endpoint
(跨进程虚拟地址无效),于是静默回退 loopback TCP(实测 ~0.78GB/s)。
唯一化后自写自读可命中 memcpy(干净集群探针实测 lo 流量归零)。
注意:memcpy 线程池 kDefaultMemcpyWorkers=1(mooncake 0.3.13 硬编码),
单流 ~1GB/s,进一步提升需改 mooncake 并重编译。
2026-10-10 16:42:00 +08:00
minke.yu 8d9bce30a7 tier realstore: 开 HC_TIMING 打点 + prefetch policy 改 wait_complete
best_effort 下低并发请求 compute 赢比赛、L3 命中不生效(mc1/2/4 实为重算);
wait_complete 后真实 L3 命中:mc1 TTFT 2989→661ms,mc8 聚合 203k→402k tok/s。
2026-10-10 15:54:11 +08:00
minke.yu 5f2ffa70b8 hicache: SGLANG_HICACHE_TIMING 打点 L3 预取全链路分阶段耗时
build-sglang-image / build (push) Failing after 1s
env 开关(默认关),开后在以下阶段输出 HC_TIMING 日志行:
query_wait/query(含 batch_exists 逐批)/hit_count all_reduce/
xfer_wait/xfer(含 batch_get 逐批)/ack_reduce/ack_drain 排队/
graft(insert_host)/prefetch_e2e/loadback。
目的:定位 L3 命中单请求 ~3s(≈重算速度)的时间分布。
真实 mooncake store 与 dummy store 实测吞吐一致,排除后端实现因素。
2026-10-10 15:29:01 +08:00
minke.yu f4ef648b16 tier L3 A/B:dummy→真实 mooncake store(线上同构 master+P2PHANDSHAKE+tcp)+ 零预热 mc 扫描脚本
dummy 后端实测单请求 L3 命中 2.96s/54k tok/s(零预热修正后),聚合 ~200k;
CPU 在 scheduler native 线程、store 守护进程与 Python 工作线程均空闲,
疑似 dummy 后端 batch_get 内部 per-key 串行。切真实 store 复核以区分
sglang L3 路径问题 vs dummy 调试后端工件。注意 bench 必须 --warmup-requests 0,
默认 warmup 会把 prompt#0 种进 L1/L2 污染 L3 测量。
2026-10-10 13:59:31 +08:00
minke.yu 6469cf4197 deploy: tier bench scripts (cp2 L1/L2/L3 matrix + R7 async aggregation) 2026-10-10 10:50:24 +08:00
minke.yu 3f8751c6ed 新增 tier-cp2 单机 compose:cp2+dspark+L2 6.4M+L3 shm dummy,测 计算/L1/L2/L3 四档 prefill 极限与异步聚合 2026-10-10 09:51:09 +08:00
minke.yu e4f1310d6f l3dummy: hicache-ratio 1.0→4.0(L2=3.2M token/P)——#38348 bench 显示收益被 L2 容量掩盖(host_capacity 拒绝 0.75M~2.97M token/P),放大 L2 验证热轮命中不被容量拒、L3 hit 嫁接后在 L2 留存 2026-10-10 09:14:07 +08:00
minke.yu a42ab6fce0 测试适配 #38348 新语义:SWA 双层缺失的 backuped 节点现在是合法匹配边界
build-sglang-image / build (push) Failing after 1s
旧语义(host_hit=0 + branching=9 + 重插建窗)被 fail-soft 取代:
FULL 段直接 host 命中(host_hit=leaf_len、branching=None),load-back
落 FULL 后由续写请求的 insert overlap 从 Full KV 重建窗口 SWA。
重写 test_swa_branching_seqlen_uses_host_full_hit 验证新链路:
match 命中 → load-back → 插入扩展 token 触发 overlap 重建 → rematch
设备全命中且洞口节点的窗口 SWA 已恢复。
2026-10-10 07:42:44 +08:00
minke.yu fa8ad18b5c 手工移植上游 #38348:hicache SWA prefetch fail-soft 降级
build-sglang-image / build (push) Failing after 1s
SWA 池 L3 取回短缺时不再整批丢弃 prefetch 结果:保留已取回的 Full KV
前缀,仅丢弃失败的 SWA 池(释放其 host buffer、从 comp_xfers 摘除),
load-back 时经既有 SWARebuild/RecoverSWAWithLockedFull 路径从 Full KV
重建窗口。Mamba 等不可重建池短缺时仍走原 all-or-nothing 整批丢弃。

配套改动(同 PR):
- swa.py validator:SWA 双层缺失但节点已 backuped(Full KV 在 host)时
  放行作为匹配边界;LOAD_BACK 收集遇双层缺失空洞时 break 而非 assert。
- prefetch_from_storage:用完整匹配前缀重算 last_hash,修复 anchor 节点
  无 host_value(异步 backup 未完成/降级嫁接)时哈希链锚错位。
- 适配差异:本树有 sidecar_hash_values 重构,belief 自愈沿用
  invalidation_hashes + trimmed_pages 语义;ongoing_prefetch 以
  CacheRequestHandle 为键,comp_xfers 改为显式参数传入检查函数。
- 单测同步更新为 fail-soft 语义(TP 一致性降级、失败池释放无泄漏、
  LOAD_BACK 锚定空洞停止收集)。
2026-10-10 07:23:15 +08:00
minke.yu 7f1edbba80 cherry-pick 上游 #40429:FlashMLA sched meta 超 48KiB smem 时跳过预计算(encoder/decoder replay 的大 batch replay pass 会触发 smem 溢出崩溃)
build-sglang-image / build (push) Failing after 1s
2026-10-10 01:20:15 +08:00
minke.yu 99c317b131 cherry-pick 上游 #42716(运行时部分):encoder SWA bounded replay 下放行 host 层前缀命中——request_window 形态的 SWA 不进树,校验门不应拦截 host-only 节点;rust adapter/test 两处 hunk 与本树结构不符未合(默认 python 树芯用不到)
build-sglang-image / build (push) Failing after 1s
2026-10-10 01:02:33 +08:00
minke.yu 2b2ede8211 l3dummy 定型为 shm+best_effort(唯一正收益组合 +26.4%);wait_complete 因 discard 白等 -17.7% 弃用 2026-10-10 00:09:02 +08:00
minke.yu b4dbcebbec mooncake standalone(dummy) 模式打通 dsv41 unified cache:放宽 standalone 检查对 LogicalHostPool 锚点的误拦(实体 buffer 由旁池运行时 register_buffer);新增 l3dummy compose(单 store 服务 64GB shm 段 + fake-cargo 跳过 rustup 下载)。实测 24x160k 热/冷 +26.4%(vs TCP best_effort -4.4%)
build-sglang-image / build (push) Failing after 1h10m35s
2026-10-10 00:06:32 +08:00
minke.yu 30ba5818e7 l3test: prefetch 策略 wait_complete→best_effort(实测等待阻塞净亏 7-17%,best_effort 持平 -4.4%)
build-sglang-image / build (push) Failing after 1s
2026-10-09 20:30:46 +08:00
minke.yu c9bfb24bcc l3 模式:默认 48×160k + 冷热轮间 120s 备份排空 2026-10-09 19:31:45 +08:00
minke.yu 3d109d50be fix: l3test L2 改 ratio 1.0(0.05 会噎死 L3 写入路径) 2026-10-09 19:30:53 +08:00
minke.yu 4eabf5e228 G9 P0: loads 排队预估字段 + /query_cache_hit 命中查询端点
A. 负载预估数据源(决策②总输入长度口径):
- LoadSnapshot 新增 queued_total_input_tokens / drain_rate_tokens_per_s
- load_inquirer 复用 pending_token_queues 求和;scheduler 60s 滑动窗
  记录每请求最终 prefill chunk 完成事件(batch_result_processor 挂钩
  set_prefill_finished_time 处,恰好每请求一次)
- /v1/loads 与 PUB socket 同源自动暴露,gateway LoadMonitor 零改动消费

B. /query_cache_hit(决策④放 D):
- io_struct 新增 QueryCacheHitReqInput/Output IPC 类型
- scheduler.handle_query_cache_hit:tree_cache.match_prefix(L1/L2)
  + query_storage_hit_length(共享 L3,逻辑镜像 decode_hicache_mixin)
- tokenizer_control_mixin:fan-out 全部 DP rank 取最优命中
- serving_chat.query_cache_hit:复用 _convert_to_internal_request +
  _tokenize_one_request 保证 chat 模板/分词与真实路径一致;含图请求
  v1 拒绝(multimodal 400);http_server 注册 /query_cache_hit
2026-10-09 18:48:27 +08:00
minke.yu 7d67d57804 fix: l3test 变体改用 hicache-ratio 0.05(V4 不支持 hicache-size) 2026-10-09 17:55:02 +08:00
minke.yu 491989284f deploy: 3p1d-l3test 变体(P 缩 L1/L2 强制 L3 命中) 2026-10-09 17:26:04 +08:00
minke.yu 53ea8fb2a7 report: 增加 input tps 列 2026-10-09 17:02:23 +08:00