Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d5a998f346 | ||
|
|
a3401b3e65 | ||
|
|
01087e689c | ||
|
|
0b45c07637 |
@@ -1,25 +0,0 @@
|
||||
# L3 SSD image: recipe-ns engine + official mooncake 0.3.14 (CUDA 13, cp312).
|
||||
#
|
||||
# Build context: this directory plus the 0.3.14 wheel
|
||||
# mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
# Use build-l3ssd-image.sh, which downloads and verifies the wheel.
|
||||
#
|
||||
# Base tag recipe-ns-433257c7b-baked-20261114 = b300/integration @ 433257c7b
|
||||
# (recipe namespace/chat-ns + gateway binary baked). Rebuild it from that commit
|
||||
# if the local tag is missing.
|
||||
FROM sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-433257c7b-baked-20261114
|
||||
|
||||
COPY mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/
|
||||
RUN python3 -m pip install --no-deps --force-reinstall \
|
||||
/tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
|
||||
&& rm -f /tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
|
||||
# The wheel's master binary uses $ORIGIN/../mooncake_transfer_engine_cuda13.libs;
|
||||
# keep it in place and expose a wrapper at the legacy path used by the manifests.
|
||||
&& printf '%s\n' \
|
||||
'#!/bin/sh' \
|
||||
'export LD_LIBRARY_PATH=/opt/sglang/lib/python3.12/site-packages/mooncake_transfer_engine_cuda13.libs:$LD_LIBRARY_PATH' \
|
||||
'exec /opt/sglang/lib/python3.12/site-packages/mooncake/mooncake_master "$@"' \
|
||||
> /opt/sglang/bin/mooncake_master \
|
||||
&& chmod +x /opt/sglang/bin/mooncake_master \
|
||||
&& python3 -c "import importlib.metadata as m; print('mooncake:', m.version('mooncake-transfer-engine-cuda13'))" \
|
||||
&& /opt/sglang/bin/mooncake_master --version | head -1
|
||||
@@ -1,139 +0,0 @@
|
||||
# B300 L3 Mooncake SSD 基线部署(2026-11-15)
|
||||
|
||||
> 以同事在 `ds41-gist-exp` 上验证的 **L3 Mooncake SSD** 配置为基础,使用我们的
|
||||
> `b300/integration` 引擎代码 + 官方 **mooncake 0.3.14**,不再依赖 hostPath
|
||||
> `/opt/mooncake-latest`。
|
||||
|
||||
## 1. 版本要求(重要)
|
||||
|
||||
- **mooncake `0.3.14`**(`mooncake-transfer-engine-cuda13`,CUDA 13 / cp312 wheel)。
|
||||
- 镜像内原为 `0.3.13.post1`;0.3.14 含上游 SSD offload 相关修复/增强。
|
||||
- wheel 与 sha256 见 `requirements-mooncake.txt`,安装必须 `--no-deps`(基底已有运行时依赖)。
|
||||
- 引擎:`b300/integration` @ `433257c7b`(recipe namespace/chat-ns + 新 gateway)。
|
||||
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
|
||||
(40.1GB,已 `ctr -n k8s.io images import`;本地 docker tag `ymkymx/sglang:...` 同名)。
|
||||
- 构建/装载:`./build-l3ssd-image.sh`(下载校验 wheel → docker build → ctr import)。
|
||||
|
||||
镜像内 `/opt/sglang/bin/mooncake_master` 是 wrapper:设置 wheel `.libs` 的
|
||||
`LD_LIBRARY_PATH` 后 exec 包内 master(已验证 `0.3.14 (git e1ea5d3)`)。
|
||||
|
||||
## 2. L3 SSD 配置
|
||||
|
||||
`mooncake-store-l3ssd.json`:
|
||||
|
||||
```json
|
||||
{"master_server_address":"127.0.0.1:50051","metadata_server":"P2PHANDSHAKE",
|
||||
"global_segment_size":"200gb","protocol":"tcp",
|
||||
"enable_ssd_offload":true,"ssd_offload_path":"/data/ymk/mooncake-ssd"}
|
||||
```
|
||||
|
||||
- 部署时把该文件放到节点 `/data/ymk/ds41/mooncake-store-l3ssd.json`,SSD 目录
|
||||
`/data/ymk/mooncake-ssd` 需存在。
|
||||
- `MOONCAKE_OFFLOAD_*`:`bucket_storage_backend` / 512GiB bucket 上限 / `fifo` /
|
||||
`flush_partial=0`(见 manifest)。
|
||||
- master 参数必须带
|
||||
`--enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false`。
|
||||
|
||||
## 3. manifest
|
||||
|
||||
| 文件 | 用途 |
|
||||
|---|---|
|
||||
| `ds41-l3ssd-test-baked-20261115.yaml` | 新建 cw `ds41-l3ssd-test`(推荐先做基线验证) |
|
||||
| `ds41-gist-exp-l3ssd-baked-20261115.yaml` | 原名 `ds41-gist-exp`,用于直接更新原栈 |
|
||||
|
||||
已通过 `kmctl create/update --dry-run` 校验。启动前先停掉占用 8 卡的 stack
|
||||
(`ds41-recipe-ns-test` 或旧 `ds41-gist-exp`),确认 GPU 归零:
|
||||
|
||||
```bash
|
||||
kmctl create cw ds41-l3ssd-test --node-group B300 \
|
||||
-f deploy/b300-k8s/ds41-l3ssd-test-baked-20261115.yaml
|
||||
# 或替换原栈:
|
||||
# kmctl update cw ds41-gist-exp -f deploy/b300-k8s/ds41-gist-exp-l3ssd-baked-20261115.yaml
|
||||
```
|
||||
|
||||
端口/角色与旧 `ds41-gist-exp` 一致:master 50051、P 30000、D 30300、gateway 30400。
|
||||
启动约 10-15 分钟;核对 P/D `max_total_num_tokens` 和 master 日志中的 SSD offload 初始化。
|
||||
|
||||
## 4. 与上一版配置的差异
|
||||
|
||||
P/D 的 `sglang serve` 参数**完全一致**;差异只在 mooncake:
|
||||
|
||||
| 项 | 旧(11-09) | 本基线 |
|
||||
|---|---|---|
|
||||
| mooncake | 镜像内 0.3.13.post1 | 镜像内 **0.3.14** |
|
||||
| master | `/opt/sglang/bin/mooncake_master`,无 offload | 同路径(0.3.14 wrapper),带 offload 参数 |
|
||||
| P/D PYTHONPATH | 无 | 无(0.3.14 已烘入) |
|
||||
| L3 配置 | 纯 TCP,无 SSD | `enable_ssd_offload` + `ssd_offload_path` |
|
||||
| `MOONCAKE_OFFLOAD_*` | 无 | bucket backend / 512GiB / fifo / flush_partial=0 |
|
||||
| TMPDIR/XDG/HF | 默认/`/data/ymk` | `/data/ymk/l3ssd/{master,p,d}-*` |
|
||||
|
||||
## 5. 注意
|
||||
|
||||
- 同事原配置使用 `SGLANG_TRACE_LOOP_SAMPLE=1.0`(全量 detail tracing,约 2% 吞吐 +
|
||||
p95 开销)。本基线为对齐其配置保留 1.0;正式上线建议改回 `0.05`。
|
||||
- `global_segment_size=200gb` + 512GiB SSD offload:上线前按宿主内存总账复核。
|
||||
- SSD 目录增长需监控(`/data` 当前约 7.6T 可用)。
|
||||
- 旧镜像 `recipe-ns-433257c7b-baked-20261114` 与 `recipe-ns-a7c668a93-baked-20261114`
|
||||
保留作回滚/对照。
|
||||
|
||||
## 6. Tokenizer A/B(2026-11-15,c580 / 1200s)
|
||||
|
||||
同一 `ds41-gist-exp`(L3 SSD + mooncake 0.3.14)整栈重启,只切 P/D
|
||||
`--tokenizer-backend`;AgentX c580 / 1200s / warmup 256 / seed 42,经 30400。
|
||||
|
||||
| 臂 | tokenizer | rps | in tok/s | out tok/s | TTFT p50/p95/p99 | ITL p50/p95/p99 | requests |
|
||||
|---|---|---:|---:|---:|---|---|---:|
|
||||
| A | `huggingface`(默认) | 23.789 | 2,257,661 | 22,224 | 2.043 / 4.176 / 7.579 s | 8.02 / 14.48 / 20.91 ms | 29,498 |
|
||||
| B | **`fastokens`** | **24.045** | **2,289,885** | **22,475** | **1.746 / 3.676 / 6.820 s** | 8.13 / 11.65 / 14.87 ms | 29,816 |
|
||||
|
||||
- 结论:fastokens 臂 **rps +1.1%**、TTFT p50/p95/p99 **−14.5% / −12.0% / −10.0%**;
|
||||
ITL p95 的差异大概率是 run-to-run 噪声(decode 不走 tokenizer),不作为判据。
|
||||
- #48928 的分段编码已生效:容器内 `DeepSeekRecipe.from_hf_tokenizer(..., fastokens tokenizer)`
|
||||
的 `_fast_prompt_tokenizer` 非 None(`fastokens.Tokenizer`);服务端未打印
|
||||
“fastokens segmented” 日志,属日志路径差异,不影响结论。
|
||||
- **当前线上实验栈保留 fastokens 臂**;要回默认用
|
||||
`ds41-gist-exp-l3ssd-baked-20261115.yaml` 更新并整栈重启。
|
||||
- 单轮对比,c580 本身有方差;如需正式定论建议每臂 ≥3 轮。
|
||||
- SSD offload 已配置(master 报 `SSD Storage: 0 B / 2.00 TB`),本次 c580 工作集未触发
|
||||
SSD 逐出(`/data/ymk/mooncake-ssd` 约 20K),不是配置未生效。
|
||||
|
||||
## 7. 多轮验证(2026-11-15,每轮整栈重启)
|
||||
|
||||
> 之前 B2/B3 在同一实例上连跑、未重启,数据受实例状态影响;下面只列**整栈重启后**的轮次。
|
||||
> 整栈重启 = `kmctl stop` + `update`(换臂),或同臂时删除 p/d/gateway/master 四个 Pod。
|
||||
|
||||
| 臂 | 轮次 | rps | TTFT p50/p95/p99 (s) | ITL p95 (ms) | requests |
|
||||
|---|---|---:|---|---:|---:|
|
||||
| A(huggingface) | A1 | 23.789 | 2.043 / 4.176 / 7.579 | 14.48 | 29,498 |
|
||||
| A | A2 | 23.552 | 2.305 / 5.894 / 8.600 | 15.05 | 29,205 |
|
||||
| B(fastokens) | B1 | 24.045 | 1.746 / 3.676 / 6.820 | 11.65 | 29,816 |
|
||||
| B | **B2(fresh)** | **24.040** | **1.708 / 3.451 / 7.105** | 11.71 | 29,810 |
|
||||
|
||||
- **B2 满足判据:TTFT p95 = 3.451s < 5s**,且与 B1(3.676s)一致;
|
||||
A 臂 fresh 轮 p95 为 4.176 / 5.894s。
|
||||
- fresh 轮均值:A rps 23.67 / p95 5.04s;B rps 24.04 / p95 3.56s
|
||||
(B +1.6% rps、p95 −29%)。
|
||||
- 结论:保留 fastokens;之前未重启的 B2/B3(p95 5.99/6.69s)仅作历史参考、不纳入。
|
||||
|
||||
## 8. L3 SSD 4TB + offload_on_evict=false(2026-11-15)
|
||||
|
||||
- 启动清单 `ds41-gist-exp-l3ssd-fastokens-20261115.yaml` 已改:master
|
||||
`--offload_on_evict=false`;P 的 `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` /
|
||||
`MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` 改为 1 TiB/rank,4 rank 合计
|
||||
master 报 `SSD Storage: 0 B / 4.00 TB`。
|
||||
- 每次实验前必须重置 L3:stop cw → 确认 Terminated 0/0 + GPU 0 MiB →
|
||||
`bash clean_l3ssd_cache.sh`(删除 `/data/ymk/mooncake-ssd` 下的 rank_*)→ update/start。
|
||||
本次清理释放 1.2T,`/data` 可用 6.4T→7.5T。
|
||||
- c580(1200s,warmup 256,seed 42,fastokens,干净 L3):
|
||||
rps 24.003、in 2.285M tok/s、out 22,447、TTFT p50/p95/p99 1.768/3.500/7.549s、
|
||||
ITL 8.15/11.74/15.31ms、29,764 请求、0 错误、valid。
|
||||
对比 B1/B2:24.045/1.746/3.676/6.820 与 24.040/1.708/3.451/7.105——
|
||||
吞吐与尾延迟均在方差内,**没有可归因于 offload_on_evict/4TB/清理的收益或回归**。
|
||||
- **重要:`offload_on_evict=false` 并没有停止 SSD 写入**。run 后 master
|
||||
`SSD Storage: 533.81 GB / 4.00 TB`,`du /data/ymk/mooncake-ssd = 535G`;
|
||||
这是 `--hicache-write-policy write_through` 的 backup 写入,和内存 eviction 无关。
|
||||
若要真正避免 SSD 写入,需要改 `write_back` 或关闭 `--enable_offload`,但那会改变
|
||||
L3 语义,需另做 A/B。
|
||||
- run 后 P cgroup `memory.current=2,198,904,565,760` ≈ 2047.5 GiB,已顶到 2 TiB
|
||||
limit;宿主 MemAvailable 1.2 TiB。下一次实验前必须先停/清/重启;不要在同一实例
|
||||
上连续跑,否则 L3 与 cgroup 都处于顶格状态。
|
||||
@@ -1,10 +0,0 @@
|
||||
# 2026-11-17 c580 基线:D tokenizer 16 + fastokens
|
||||
|
||||
- manifest: `manifest.yaml`(= `ds41-gist-exp-20261117-dtok16-fastokens.yaml`)
|
||||
- P: cache / 无 optimistic / tokenizer-worker 16 / fastokens / hicache ratio 4 / L3 SSD 4T
|
||||
- D: tokenizer-worker 16 / fastokens
|
||||
- Gateway: 正常(无 P admission cap,health/circuit 开启,decode-max-running 512)
|
||||
- L3:每轮前清空;完整重启
|
||||
- c580 结果:rps 24.045,in 2.290M tok/s,out 22.449k tok/s,
|
||||
TTFT 1.743/3.772/6.580 s,ITL p95 10.74 ms,29,816 req,valid=true。
|
||||
- 原始结果:`experiments/agentx-align/results-c580-dtok16-fastokens-20261117/`
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1 +0,0 @@
|
||||
{"rps": 24.04512015569562, "in_tput": 2290117.7612668574, "out_tput": 22449.32127276801, "ttft_p50": 1743.335193, "ttft_p95": 3771.88552025, "ttft_p99": 6579.86958935, "itl_p50": 8.162268671611512, "itl_p95": 10.741317697249674, "itl_p99": 11.806237803568294, "lat_p95": 31884.86991075, "requests": 29816.0, "isl_p50": 79909.0, "osl_p50": 339.0, "cache_pct": NaN, "valid": true}
|
||||
@@ -1,20 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build and load the L3 SSD image (mooncake 0.3.14) on b300-01.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
WHEEL=mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
URL=https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/$WHEEL
|
||||
SHA=74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
|
||||
TAG=recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
REPO=sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang
|
||||
|
||||
if [ ! -f "$WHEEL" ]; then
|
||||
curl -fL -o "$WHEEL" "$URL"
|
||||
fi
|
||||
echo "$SHA $WHEEL" | sha256sum -c -
|
||||
|
||||
docker build -t "ymkymx/sglang:$TAG" -f Dockerfile.mooncake-0.3.14 .
|
||||
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
|
||||
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
|
||||
echo "loaded $REPO:$TAG"
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Reset the mooncake L3 SSD KV cache before an experiment.
|
||||
#
|
||||
# MUST run on b300-01 only after the ds41-gist-exp workload is fully stopped:
|
||||
# 1) kmctl stop cw ds41-gist-exp --wait ...
|
||||
# 2) kubectl get cw ... -> Terminated 0/0, nvidia-smi -> 0 MiB on GPU0-7
|
||||
# 3) bash clean_l3ssd_cache.sh
|
||||
# 4) kmctl update/start the new manifest
|
||||
#
|
||||
# The script refuses to delete anything while a mooncake master or sglang
|
||||
# server process is still visible on the host.
|
||||
set -euo pipefail
|
||||
|
||||
SSD_DIR="${SSD_DIR:-/data/ymk/mooncake-ssd}"
|
||||
|
||||
if pgrep -af '[m]ooncake_master|[s]glang serve' >/dev/null 2>&1; then
|
||||
echo "refusing: mooncake_master or sglang serve is still running" >&2
|
||||
pgrep -af '[m]ooncake_master|[s]glang serve' >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$SSD_DIR" ]; then
|
||||
echo "SSD cache dir does not exist: $SSD_DIR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "before: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
|
||||
find "$SSD_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
|
||||
echo "after: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
|
||||
df -h /data | tail -1
|
||||
@@ -1,542 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
|
||||
\ 32 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
|
||||
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
|
||||
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,544 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
|
||||
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
|
||||
\ 16000000 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n\
|
||||
\ --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30\
|
||||
\ --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\
|
||||
\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,544 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
|
||||
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
|
||||
\ 16000000 --g9i-p-qps 20 --g9i-p-refresh-ms 250 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
|
||||
\ 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
|
||||
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
|
||||
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-l3ssd-test
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -1,41 +0,0 @@
|
||||
# Dynamo-enabled B300 serving image.
|
||||
#
|
||||
# The sglang runtime tree inside the base image is byte-identical to
|
||||
# b300/dynamo @ 8b67636010 (the only commits between the baked tree
|
||||
# 433257c7b0 and 8b67636010 are deploy/docs), so this layer only swaps the
|
||||
# preinstalled Dynamo 1.6.0.dev snapshot for the released 1.6.2 runtime.
|
||||
#
|
||||
# Build:
|
||||
# docker build -t ymkymx/sglang:dynamo-8b6763601-baked-20261117 .
|
||||
# or use ./build-dynamo-image.sh (also imports into containerd).
|
||||
|
||||
ARG BASE=ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
FROM ${BASE}
|
||||
|
||||
ARG DYNAMO_VERSION=1.6.2
|
||||
ARG PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
|
||||
LABEL ai.sglang.build.commit="8b67636010e54fc96e5aeba9b0139b38aa18b000"
|
||||
LABEL ai.dynamo.version="${DYNAMO_VERSION}"
|
||||
LABEL org.opencontainers.image.description="SGLang B300 (b300/dynamo) + NVIDIA Dynamo ${DYNAMO_VERSION}"
|
||||
|
||||
# --no-deps: the base image already carries the full runtime dependency set
|
||||
# (torch 2.14.1, ai-dynamo deps, nixl 1.5.0). Resolving would try to pull
|
||||
# sglang==0.5.19 from the `sglang` extra and downgrade torch.
|
||||
RUN pip uninstall -y ai-dynamo ai-dynamo-runtime >/dev/null 2>&1 || true \
|
||||
&& pip install --no-cache-dir --no-deps -i "${PIP_INDEX}" \
|
||||
"ai-dynamo==${DYNAMO_VERSION}" "ai-dynamo-runtime==${DYNAMO_VERSION}"
|
||||
|
||||
# CPU-only smoke: both CLIs must import the sglang args we build from source.
|
||||
RUN python - <<'PY'
|
||||
import importlib.metadata as m
|
||||
import sglang
|
||||
import dynamo.sglang # noqa: F401
|
||||
import dynamo.frontend # noqa: F401
|
||||
print("dynamo", m.version("ai-dynamo"), "runtime", m.version("ai-dynamo-runtime"))
|
||||
print("sglang", sglang.__file__)
|
||||
PY
|
||||
RUN python -m dynamo.sglang --help >/dev/null && python -m dynamo.frontend --help >/dev/null
|
||||
|
||||
ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
|
||||
CMD []
|
||||
@@ -1,216 +0,0 @@
|
||||
# Dynamo 支持(b300/dynamo)
|
||||
|
||||
> 分支:`b300/dynamo`(基点 `8b67636010`,= `main` = `b300/integration` 的 2026-11-17 c580 基线)
|
||||
> 状态:**代码/镜像/启动脚本已就绪,静态兼容性已验证;GPU 运行时验证待窗口**
|
||||
> (2026-11-17 时 `ds41-gist-exp` 占满 8 卡)。
|
||||
|
||||
本目录是 B300 线切换到 **NVIDIA Dynamo** 的入口:Dynamo frontend + 我们的 sglang
|
||||
worker(P/D 分离)。它与现有 `ds41-gist-exp` 的区别:**入口 + PD 编排从 g9i gateway
|
||||
换成 Dynamo frontend(`:33000`)**;**KV 数据面仍是 sglang 原生 disaggregation(mooncake)**。
|
||||
|
||||
## 1. 兼容性结论(2026-11-17)
|
||||
|
||||
- **不需要 sglang 代码改动**。Dynamo 1.6.2 的 `dynamo.sglang` 后端通过 `sgl.Engine`
|
||||
公开接口工作,`_compat.py` 对 0.5.18/0.5.19 的兼容层在当前树(main-era)全部命中
|
||||
新版分支。
|
||||
- 用 Dynamo 1.6.2 自带单测跑了当前树(`PYTHONPATH=<tree>/python`,镜像内先装
|
||||
`pytest-asyncio`):**985 passed / 20 failed / 1 skipped**。同一镜像用自带 sglang
|
||||
nightly 跑对照也是 **985 passed / 20 failed / 1 skipped**,`diff` 后失败用例集合
|
||||
**逐条相同**(均为视频生成等既有环境相关用例)。→ 无 API / 行为漂移。
|
||||
- `python -m dynamo.sglang --help`(1398 行,sglang 参数已合并)与
|
||||
`python -m dynamo.frontend --help` 均 rc=0。
|
||||
- Dynamo 依赖的 `ServerArgs` 字段(`enable_forward_pass_metrics`、
|
||||
`forward_pass_metrics_ipc_name`、`elastic_ep_backend`、`dllm_algorithm`、
|
||||
`incremental_streaming_output`、`kv_events_config`、`allow_auto_truncate` 等)
|
||||
在当前树全部存在。
|
||||
|
||||
复现命令(CPU,容器内):
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint bash \
|
||||
-e PYTHONPATH=/data/ymk/wt-dynamo/python \
|
||||
-v /data/ymk/wt-dynamo:/data/ymk/wt-dynamo:ro \
|
||||
<dynamo-image> -c 'python -m dynamo.sglang --help >/dev/null && echo ok'
|
||||
```
|
||||
|
||||
## 2. 版本矩阵
|
||||
|
||||
| 组件 | 版本 | 备注 |
|
||||
|---|---|---|
|
||||
| sglang | `b300/dynamo` @ `8b67636010` | 与镜像内已烘源码 `433257c7b0` 的 `python/sglang` **逐字节一致**(中间只有 deploy/docs 提交) |
|
||||
| Dynamo | **1.6.2**(`ai-dynamo` + `ai-dynamo-runtime`) | 基底镜像原本带 `1.6.0.dev20261019`,Dockerfile 覆盖 |
|
||||
| nixl | 1.5.0(`nixl-cu13` 1.5.0) | 基底已带;Dynamo 的 SGLang 后端**不用**它做 KV 传输 |
|
||||
| torch | 2.14.1+cu130 | 基底已带,不因 Dynamo 变化 |
|
||||
| sglang-kernel | 0.4.10 | 基底已带,与当前树 pin 一致 |
|
||||
| mooncake | 0.3.14(L3 SSD) | 基底已带(见 `../README.md`) |
|
||||
|
||||
- KV 传输仍是 **sglang 原生 disaggregation + mooncake**;`--connector nixl` 只对
|
||||
vLLM/TRT-LLM 生效,对 SGLang 是 no-op(10-26 已确认,本脚本不带它)。
|
||||
- NVLink:`MC_INTRANODE_NVLINK=true` + `MC_INTRA_NVLINK=true` + `--pid host` +
|
||||
宿主已烘的 NVLink host 修复。
|
||||
|
||||
### 2.1 PD 的两层要分开看(常被混淆)
|
||||
|
||||
| 层 | 由谁实现 | 在本部署里的表现 |
|
||||
|---|---|---|
|
||||
| **PD 编排 / 路由**:选 P、选 D、把 `bootstrap_info` 从 P 传给 D、流式回传 | **Dynamo 自己的 Rust runtime + frontend PrefillRouter**(不是 sglang-router/mini_lb) | `dynamo.frontend :33000`;Dynamo 的 `sglang` handler 只负责把 bootstrap 信息塞进/取出 `Engine.async_generate` |
|
||||
| **KV 数据面**:bootstrap 握手后的 KV 传输 | **sglang 原生 disaggregation**(`--disaggregation-mode` + `--disaggregation-transfer-backend mooncake`) | P/D 日志的 `Using Intra-Node NVLink transport`;10-26 的 NVLink host 补丁仍然生效 |
|
||||
|
||||
Dynamo 的 `--connector nixl` / `--kv-transfer-config` 只对 vLLM/TRT-LLM 生效(vLLM 需要
|
||||
`--kv-transfer-config '{"kv_connector":"NixlConnector",...}'`);SGLang 后端把
|
||||
`bootstrap_host/port/room` 直接传给 `Engine.async_generate`,传输由 sglang 的 bootstrap
|
||||
server + mooncake 完成(`prefill_handler.py` 产出 `disaggregated_params`,
|
||||
`decode_handler.py` 消费 `bootstrap_info`)。所以 P/D 两侧仍然必须写
|
||||
`--disaggregation-mode`,且 `--disaggregation-transfer-backend` 要与 sglang 侧一致。
|
||||
|
||||
> 结论:**编排是 Dynamo 的,KV 线是 sglang 的**。两者不是同一套,但也不是各走各的——
|
||||
> Dynamo 控制面负责“谁和谁配对”,sglang 数据面负责真正搬 KV。
|
||||
|
||||
### 2.3 Dynamo 1.6.2 的 tokenizer / chat processor(实测)
|
||||
|
||||
DeepSeek-V4.1 没有 HF `chat_template`,默认的 Rust processor 无法渲染;需要用
|
||||
SGLang 的 pre/post processor:
|
||||
|
||||
```
|
||||
--dyn-chat-processor sglang \
|
||||
--dyn-preprocess-workers 8 \
|
||||
--tokenizer fastokens --trust-remote-code \
|
||||
--tool-call-parser deepseekv41 \
|
||||
--reasoning-parser deepseek-v41
|
||||
```
|
||||
|
||||
本次踩通/修掉的四个点:
|
||||
|
||||
1. **缺 `enencoding_dsv4`**:`--dyn-chat-processor sglang` 会 import
|
||||
`sglang.srt.entrypoints.openai.enencoding_dsv4`;我们的 fork 用 deepseek-recipe,没有该模块,
|
||||
chat 直接 500。已在 `b300/dynamo` 加 shim `enencoding_dsv4.py`(基于 recipe 的
|
||||
`render_conversation`),token ids 与生产 `DeepSeekRecipe.encode` 逐 token 相同。
|
||||
2. **`--tokenizer fastokens` 不会自动传给 SGLang processor**:该 flag 只作用于 Dynamo 的
|
||||
Rust model-card tokenizer;`sglang_processor._load_tokenizer()` 调 `get_tokenizer()`
|
||||
时默认 `huggingface`。需要把 `FrontendConfig.tokenizer_backend` 传进去(FE hostPath 补丁),
|
||||
日志出现 `fastokens backend enabled - transformers patched successfully` 才算生效。
|
||||
3. **v41 thinking 解析**:Dynamo 的 `_THINKING_OPT_IN` 缺 `deepseek-v41`,显式
|
||||
`chat_template_kwargs.thinking=true` 时 `reasoning_content` 不会拆出来。已补上;
|
||||
默认请求仍是 chat(worker 没有发布 `default_thinking_mode`;生产默认 thinking 需要
|
||||
register 侧补丁,TODO)。
|
||||
4. **`--gateway-workers N`(worker 侧)** 是另一条路:给默认 Rust processor 用,启动 N 个
|
||||
tokenizer/gateway 进程并设置 SGLang `--tokenizer-worker-num`;`--use-sglang-tokenizer`
|
||||
已 deprecated。本次走 frontend processor,不需要它。
|
||||
|
||||
另:`encoding_dsv4`/`thinking` 的 FE 补丁目前以 hostPath 挂载在容器里,尚未烘进镜像;
|
||||
重建镜像时要一起带上(见 `start_dynamo_pd.sh` 的 `-v` 列表)。
|
||||
|
||||
### 2.4 2026-11-17 首次运行结果(4P(dp4)+4D(dp4))
|
||||
|
||||
- 启动约 7 分钟;P `max_total_num_tokens=16,157,696`、D `58,859,520`(与 11-17
|
||||
`ds41-gist-exp` 基线一致);P/D 均出现 `Using Intra-Node NVLink transport`。
|
||||
- chat / tool_calls / reasoning_content 均可;prefix cache 正常(16k prompt 第二轮
|
||||
cached 15,872;80k 第二轮 cached 79,872)。
|
||||
- 32 并发 × ~900 token prompt(85% 命中,warm):**wall 0.62 s,32/32 成功**;
|
||||
16k 冷 0.47 s / 热 0.24 s;80k 热 0.25 s(冷 80k ~1.04 s,HF tokenizer 时的数字)。
|
||||
**重启后前 1–2 波会明显慢(8–9 s)是 warmup 伪影,benchmark 必须先热身。**
|
||||
- P 的 OTel batch-span link 会报 `expected bytes, TraceNullContext found`(旧镜像);
|
||||
已在分支修 `batch_telemetry.py`,本次运行用 `SGLANG_BATCH_SPAN=0` 规避。
|
||||
|
||||
### 2.2 Dynamo 不会消除 sglang 的 bootstrap / 握手尾巴
|
||||
|
||||
Dynamo 换掉的是控制面,sglang 内部的 bootstrap 路径原样保留:
|
||||
|
||||
- P 侧 `PrefillBootstrapQueue.pop_bootstrapped()` 与 `resolve_waiting_queue_bootstrap()`
|
||||
仍然在 scheduler 每个 pass 里做 `poll_and_all_reduce_attn_cp_tp_group(...)`
|
||||
(`disaggregation/prefill.py:495-556`、`638-675`);bootstrap 完成只在 pass 边界被发现。
|
||||
- D 侧 `DecodePreallocQueue` / `_ensure_prefill_info` / 传输接收队列也一样。
|
||||
- 因此 `prefill_bootstrap`(P queue entry → bootstrap done)与 `decode_transferred`
|
||||
(等 P 完成 + 收 KV)的口径不变,Dynamo frontend 必须等它们完成才会开始 decode;
|
||||
长 pass / DP skew / 大 batch 会按 pass 放大这个延迟。
|
||||
|
||||
实测参考(当前栈):c620 下 `prefill_bootstrap` p50 0.23–0.36 s、p95 1.66 s(fresh)/ 5.05 s
|
||||
(不重启),而 `mooncake_worker_send_session` p50 只有 2.6 ms —— 卡的是握手/轮询,不是搬 KV。
|
||||
Dynamo 的收益在入口/路由/可观测性;要压 bootstrap 尾部必须改 sglang scheduler 的轮询/预分配路径
|
||||
(或先用 optimistic prefill + `hicache-write-policy write_back` 把 prefill 与握手重叠)。
|
||||
|
||||
## 3. 构建镜像
|
||||
|
||||
```bash
|
||||
cd deploy/b300-k8s/dynamo
|
||||
./build-dynamo-image.sh # 构建 + ctr import
|
||||
# IMPORT=0 只构建;TAG=... 自定义 tag
|
||||
```
|
||||
|
||||
默认产物:`ymkymx/sglang:dynamo-8b6763601-baked-20261117`
|
||||
(= `sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:...`,同一 image ID)。
|
||||
|
||||
要点:Dynamo 必须 `pip install --no-deps`。基底已带全部运行时依赖;一旦让 pip 解析,
|
||||
`ai-dynamo` 的 `sglang` extra 会试图装 `sglang==0.5.19` 并降级 torch。
|
||||
|
||||
## 4. 启动(docker,4P(dp4)+4D(dp4))
|
||||
|
||||
先停掉占卡的工作负载并确认 GPU 0-7 为 0 MiB,再:
|
||||
|
||||
```bash
|
||||
./start_dynamo_pd.sh
|
||||
./smoke.sh # /v1/models + /v1/chat/completions
|
||||
tail -f /data/ymk/gist-dsv41/dyn-{fe,p,d}.log
|
||||
```
|
||||
|
||||
端口:frontend **33000**、P **30000**、D **30300**;P/D 的 sglang 指标仍在原端口,
|
||||
Dynamo 自己的指标在 frontend。
|
||||
|
||||
> ⚠️ 启动耗时与普通 P/D 栈相同(P/D 各 ~6-10 分钟;`/root/.cache/sglang` 挂载可复用
|
||||
> autotune/JIT 缓存)。**先等 P/D ready 再开压测**:Dynamo 有冷启动首波长尾
|
||||
> (空闲后首波 3–22s,暖机后消失)。
|
||||
|
||||
## 5. 与 2026-10-26 Dynamo 试验的差异
|
||||
|
||||
| 项 | 10-26 试验 | 本目录 |
|
||||
|---|---|---|
|
||||
| Dynamo | runtime 镜像 `sglang-runtime:1.6.0` | PyPI **1.6.2**(烘进 B300 镜像) |
|
||||
| 模型 | `DeepSeek-V4.1-Flash` | `DeepSeek-V4.1-Flash-1017`(served name 保持原路径) |
|
||||
| 并行 | tp4 + tp4(`--moe-a2a-backend none`) | **dp4 + dp4**(dp-attention + dp-lm-head + megamoe + flashmla_mega) |
|
||||
| 入口 | `dynamo.frontend :33000` | 同上 |
|
||||
| 缓存 | L2/L3 32GB | 当前生产 L3 SSD 4TB + `write_through` |
|
||||
| 门控 | `SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0` | 同(依赖树内 `7f6fe22445`/`ee9170e7bb` fail-soft) |
|
||||
|
||||
dp4 是 DSpark PD 的硬要求:两侧 `attn_tp_size` 必须相等。dp4 下 attn_tp = tp/dp = 1,
|
||||
P/D 一致;若一侧 tp4、另一侧 dp4,`disaggregation/common/conn.py:1355` 会直接拒绝
|
||||
(`DSpark PD requires the same TP size on both servers`)。
|
||||
|
||||
## 6. 已知约束 / 坑
|
||||
|
||||
1. **`--host 0.0.0.0` 必须**:sglang bootstrap server 跟随 `--host`;写 127.0.0.1 时
|
||||
D 侧拿不到 prefill parallel info → `HTTP 500: Decode handshake failed`。
|
||||
2. **`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python`**:baked 镜像没有
|
||||
`rust_tree_core` `.so`(fake-cargo 阻止现场编译);1M 上下文下 unified cache 会选
|
||||
Rust TreeCore 并 `ModuleNotFoundError`。本脚本已设为 `python`。
|
||||
3. **Dynamo 1.6.2 安装用 `--no-deps`**:见 §3。`aisimulate` 在清华源只有
|
||||
`0.13.0.dev2`(Dynamo 1.6.2 元数据要 `0.13.0`),实测 import 正常。
|
||||
4. **`--dyn-preprocess-workers`** 取代了旧报告的 “gateway workers”:1.6.2 默认
|
||||
auto(每个 model 最多 2),可显式设 4–8 提升高并发 tokenize 吞吐;仅在
|
||||
`--dyn-chat-processor sglang` 下生效。
|
||||
5. **Dynamo frontend 没有 g9i 的准入闸门**(`--g9i-p-queue-reqs-cap` 等)。Dynamo 自己的
|
||||
flow control(`--router-queue-threshold` / `--load-aware`)在 10-26 实测无效,
|
||||
有效闸门仍是客户端/网关并发上限。
|
||||
6. **P/D 的 health / readiness**:`dynamo.sglang` worker 通过 discovery 注册,不使用
|
||||
sglang 的 `/health` 作为就绪判据;k8s 化时探针应指向 frontend,或改用 dynamo 的
|
||||
health payload(`dynamo.sglang.health_check`)。
|
||||
7. `--enable-trace` 的 OTLP 服务名分别为 `dyn-p` / `dyn-d`(与旧试验一致);
|
||||
Jaeger `:4317` 可用。
|
||||
8. 端口分配遵守 `30000-32767` 固定端口纪律;frontend 33000 与现有栈不冲突。
|
||||
|
||||
## 7. 待办
|
||||
|
||||
- [ ] **GPU 冒烟**:等 `ds41-gist-exp` 停/窗口;验证 PD 全链路、NVLink transport 日志
|
||||
(`Using Intra-Node NVLink transport`)、DSpark accept len、L2/L3 命中。
|
||||
- [ ] **性能基线**:c580/1200s(同 `baselines/20261117-dtok16-fastokens` 口径)对比
|
||||
g9i gateway 栈;关注 Dynamo frontend 的 tokenize 开销与冷启动长尾。
|
||||
- [ ] **k8s cw manifest**:`ds41-dynamo`(frontend + p + d + mooncake-master),
|
||||
探针/服务注册按 Dynamo discovery 设计;本目录先提供 docker 形态。
|
||||
- [ ] 视需要把 namespace / Chat-Completions 工具展平(`b300/feat-recipe-ns` 已进
|
||||
`b300/integration`)在 Dynamo frontend 侧的行为纳入 `preflight/issue-set`。
|
||||
|
||||
## 8. 参考
|
||||
|
||||
- 旧运行记录:`D:\B300\experiments\dsv41-gist-deepinfer\REPORT-DYNAMO-20261026.md`
|
||||
- 启动脚本(旧,tp4):`D:\B300\experiments\dsv41-gist-deepinfer\start_dynamo_pd.sh`
|
||||
- Dynamo 仓库(本地):`D:\B300\dynamo`(1.6.0 源码树,仅作 API 参考;实际运行用 1.6.2 wheel)
|
||||
- L3 SSD 基线:`deploy/b300-k8s/README.md`
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build (and optionally load into containerd) the Dynamo-enabled B300 image.
|
||||
#
|
||||
# ./build-dynamo-image.sh # build + ctr import
|
||||
# IMPORT=0 ./build-dynamo-image.sh # build only
|
||||
# TAG=my-tag ./build-dynamo-image.sh # custom tag
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
BASE=${BASE:-ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115}
|
||||
TAG=${TAG:-dynamo-8b6763601-baked-20261117}
|
||||
REPO=${REPO:-sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang}
|
||||
IMPORT=${IMPORT:-1}
|
||||
DYNAMO_VERSION=${DYNAMO_VERSION:-1.6.2}
|
||||
|
||||
echo "base=$BASE tag=$TAG dynamo=$DYNAMO_VERSION"
|
||||
docker build \
|
||||
--build-arg BASE="$BASE" \
|
||||
--build-arg DYNAMO_VERSION="$DYNAMO_VERSION" \
|
||||
-t "ymkymx/sglang:$TAG" \
|
||||
-f Dockerfile .
|
||||
|
||||
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
|
||||
|
||||
docker run --rm --entrypoint bash "ymkymx/sglang:$TAG" -c '
|
||||
python - <<PY
|
||||
import importlib.metadata as m, importlib.util
|
||||
import dynamo.sglang, dynamo.frontend # noqa: F401
|
||||
print("ok: ai-dynamo", m.version("ai-dynamo"), "| sglang", importlib.util.find_spec("sglang").origin)
|
||||
PY
|
||||
python -m dynamo.sglang --help >/dev/null
|
||||
python -m dynamo.frontend --help >/dev/null
|
||||
'
|
||||
|
||||
if [ "$IMPORT" = "1" ]; then
|
||||
echo "importing $REPO:$TAG into containerd (k8s.io) ..."
|
||||
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
|
||||
fi
|
||||
echo "done: $REPO:$TAG"
|
||||
@@ -1,15 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dynamo frontend smoke test.
|
||||
# ./smoke.sh [base_url] [model_name]
|
||||
set -euo pipefail
|
||||
BASE=${1:-http://127.0.0.1:33000}
|
||||
MODEL=${2:-/data/models/DeepSeek-V4.1-Flash}
|
||||
|
||||
echo "== /v1/models"
|
||||
curl -fsS "$BASE/v1/models" | head -c 600; echo
|
||||
|
||||
echo "== chat"
|
||||
curl -fsS "$BASE/v1/chat/completions" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"2+3=? Answer with the number only.\"}],\"max_tokens\":64,\"temperature\":0}" \
|
||||
| head -c 1200; echo
|
||||
@@ -1,183 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# 4P(dp4)+4D(dp4) PD on NVIDIA Dynamo 1.6.2 + our B300 sglang (b300/dynamo @ 8b67636010).
|
||||
#
|
||||
# frontend :33000 (Dynamo router; replaces the g9i gateway)
|
||||
# prefill :30000 GPU 0-3, mooncake NVLink, HiCache L2 + L3 SSD
|
||||
# decode :30300 GPU 4-7
|
||||
#
|
||||
# Differences vs the 2026-10-26 Dynamo bring-up (REPORT-DYNAMO-20261026.md):
|
||||
# * dp4 (tp4 + dp-attention + dp-lm-head) on both sides instead of tp4, matching
|
||||
# the current production stack. Both sides must expose attn_tp_size==1 or the
|
||||
# DSpark PD handshake rejects the pair ("same TP size on both servers").
|
||||
# * image carries Dynamo 1.6.2 (the 10-26 run used the 1.6.0 runtime image).
|
||||
# * L3 SSD mooncake 0.3.14 + the 1017 model / fastokens tokenizer.
|
||||
#
|
||||
# NOT validated on GPU yet (2026-11-17: ds41-gist-exp owns all 8 GPUs).
|
||||
set -uo pipefail
|
||||
|
||||
IMG=${IMG:-ymkymx/sglang:dynamo-8b6763601-baked-20261117}
|
||||
MODEL=${MODEL:-/data/models/DeepSeek-V4.1-Flash-1017}
|
||||
SERVED=${SERVED:-/data/models/DeepSeek-V4.1-Flash}
|
||||
D=${D:-/data/ymk/gist-dsv41}
|
||||
STORE=${STORE:-$D/dynamo-store}
|
||||
L3_CONF=${L3_CONF:-/data/ymk/ds41/mooncake-store-l3ssd.json}
|
||||
JITCACHE=${JITCACHE:-/data/ymk/cache/sglang}
|
||||
FE_PORT=${FE_PORT:-33000}
|
||||
# Frontend tokenizer/chat processing (1.6.2):
|
||||
# --dyn-chat-processor sglang frontend does SGLang-native template rendering,
|
||||
# tokenization, tool/reasoning parsing
|
||||
# --dyn-preprocess-workers N ProcessPool of N tokenizer/template workers
|
||||
# --tokenizer fastokens fastokens BPE encoder (falls back to HF)
|
||||
# Parser names must be explicit: sglang's "auto" is resolved at server startup
|
||||
# and is not understood by the frontend's parser factory.
|
||||
PREPROCESS_WORKERS=${PREPROCESS_WORKERS:-8}
|
||||
TOOL_PARSER=${TOOL_PARSER:-deepseekv41}
|
||||
REASONING_PARSER=${REASONING_PARSER:-deepseek-v41}
|
||||
FE_ARGS=${FE_ARGS:---router-mode round-robin \
|
||||
--dyn-chat-processor sglang \
|
||||
--dyn-preprocess-workers $PREPROCESS_WORKERS \
|
||||
--tokenizer fastokens --trust-remote-code \
|
||||
--tool-call-parser $TOOL_PARSER \
|
||||
--reasoning-parser $REASONING_PARSER}
|
||||
|
||||
mkdir -p "$STORE"; chmod 777 "$STORE" 2>/dev/null || true
|
||||
|
||||
docker rm -f dyn-fe dyn-p dyn-d >/dev/null 2>&1
|
||||
sleep 3
|
||||
# Dynamo/sglang children can outlive the container when --pid host is used.
|
||||
for p in $(ps -eo pid,comm | awk '$2 ~ /^sglang::/ {print $1}'); do kill -9 "$p" 2>/dev/null; done
|
||||
sleep 2
|
||||
|
||||
# L3 store master (not part of the Dynamo graph). Same pod/port as the cw master.
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx dyn-mooncake; then
|
||||
docker rm -f dyn-mooncake >/dev/null 2>&1
|
||||
docker run -d --name dyn-mooncake --network host "$IMG" \
|
||||
/opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 \
|
||||
--eviction_high_watermark_ratio=0.90 \
|
||||
--enable_offload=true --offload_on_evict=false \
|
||||
--promotion_on_hit=false --offload_force_evict=false \
|
||||
> "$D/dyn-mooncake.log" 2>&1
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
COMMON_ENVS=(
|
||||
-e SGLANG_RAGGED_VERIFY_MODE=static
|
||||
-e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1
|
||||
-e SGLANG_DSV4_KV_LAYOUT=v41_new
|
||||
-e MC_INTRANODE_NVLINK=true
|
||||
-e MC_INTRA_NVLINK=true
|
||||
-e SGLANG_MOONCAKE_SEND_AUX_TCP=true
|
||||
# The baked image has no rust_tree_core .so; force the Python radix tree.
|
||||
-e SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
|
||||
-e SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600
|
||||
-e SGLANG_DISAGGREGATION_QUEUE_SIZE=16
|
||||
-e SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=32
|
||||
-e SGLANG_BATCH_TELEMETRY=1
|
||||
)
|
||||
P_ENVS=(
|
||||
-e SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1
|
||||
-e SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1
|
||||
-e SGLANG_OPT_DSV41_BF16_INDEX_LOGITS=1
|
||||
-e SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536
|
||||
-e SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0
|
||||
-e SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=$L3_CONF
|
||||
-e SGLANG_HICACHE_PREFETCH_IO_THREADS=4
|
||||
-e SGLANG_HICACHE_BACKUP_IO_THREADS=4
|
||||
-e SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1
|
||||
-e SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH=1
|
||||
# The baked image predates the TraceNullContext guard; keep batch spans off
|
||||
# until the next image build (histograms are unaffected).
|
||||
-e SGLANG_BATCH_SPAN=0
|
||||
)
|
||||
D_ENVS=(
|
||||
-e SGLANG_ENABLE_METRICS_DP_ATTENTION=1
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------- frontend
|
||||
docker run -d --name dyn-fe --network host --entrypoint bash \
|
||||
-v /data:/data -v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/wt-dynamo/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:ro \
|
||||
-v /data/ymk/dynamo-patch-sglang_prepost.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_prepost.py:ro \
|
||||
-v /data/ymk/dynamo-sglang_processor.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_processor.py:ro \
|
||||
"$IMG" \
|
||||
-c "python -m dynamo.frontend --discovery-backend file --http-port $FE_PORT $FE_ARGS \
|
||||
> $D/dyn-fe.log 2>&1"
|
||||
|
||||
# ---------------------------------------------------------------- prefill
|
||||
docker run -d --name dyn-p --gpus all -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
|
||||
"${COMMON_ENVS[@]}" "${P_ENVS[@]}" \
|
||||
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
|
||||
-v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
|
||||
"$IMG" python -m dynamo.sglang \
|
||||
--endpoint dyn://dynamo.prefill.generate --discovery-backend file \
|
||||
--disaggregation-mode prefill \
|
||||
--trust-remote-code --tokenizer-backend fastokens \
|
||||
--model-path "$MODEL" --served-model-name "$SERVED" \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--dyn-default-thinking-mode enabled \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-dp-attention-local-control-broadcast \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-p \
|
||||
--watchdog-timeout 3600 \
|
||||
> "$D/dyn-p.log" 2>&1
|
||||
|
||||
# ---------------------------------------------------------------- decode
|
||||
docker run -d --name dyn-d --gpus all -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
|
||||
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
|
||||
"${COMMON_ENVS[@]}" "${D_ENVS[@]}" \
|
||||
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
|
||||
-v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
|
||||
"$IMG" python -m dynamo.sglang \
|
||||
--endpoint dyn://dynamo.backend.generate --discovery-backend file \
|
||||
--disaggregation-mode decode \
|
||||
--trust-remote-code --tokenizer-backend fastokens \
|
||||
--model-path "$MODEL" --served-model-name "$SERVED" \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--dyn-default-thinking-mode enabled \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-d \
|
||||
--watchdog-timeout 3600 \
|
||||
> "$D/dyn-d.log" 2>&1
|
||||
|
||||
sleep 12
|
||||
docker ps --format '{{.Names}}\t{{.Status}}' | grep -E '^dyn-' || true
|
||||
echo "---- frontend log"; tail -3 "$D/dyn-fe.log" 2>/dev/null
|
||||
echo "---- prefill log"; tail -3 "$D/dyn-p.log" 2>/dev/null
|
||||
echo "---- decode log"; tail -3 "$D/dyn-d.log" 2>/dev/null
|
||||
echo "[$(date +%H:%M:%S)] launched: FE=$FE_PORT P=30000 D=30300 (image $IMG)"
|
||||
@@ -1,62 +0,0 @@
|
||||
<h1>当前基线:ds41-gist-exp(4T L3 SSD,2026-11-17)</h1>
|
||||
<callout emoji="📌" background-color="light-blue"><p>本节是 2026-11-17 定稿的 c580/c620 基线,后续 Dynamo 叠加以此为底座。旧的 ds41-gist-1p1d 章节已归档,不再作为当前部署。</p></callout>
|
||||
|
||||
<h2>1. 基线身份与镜像</h2>
|
||||
<ul>
|
||||
<li><b>工作负载</b>:<code>ds41-gist-exp</code> / <code>cw-dblft6sgrhagmpdt8ef0</code>,namespace <code>prj-d7te67bj0sq2fgdrmbig</code>,b300-01。</li>
|
||||
<li><b>镜像</b>:<code>sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115</code>(40.1GB,含 Mooncake 0.3.14 + L3 SSD 支持)。</li>
|
||||
<li><b>manifest</b>:<code>deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml</code>;基线副本 <code>deploy/b300-k8s/baselines/20261117-dtok16-fastokens/manifest.yaml</code>;b300 副本在 <code>/data/ymk/deploy/b300-k8s/</code>。</li>
|
||||
<li><b>形态</b>:RBG 三角色 <code>p</code> / <code>d</code> / <code>gateway</code> + <code>mooncake-master</code>;全 hostNetwork / hostPID / privileged;P=GPU0-3,D=GPU4-7。</li>
|
||||
</ul>
|
||||
|
||||
<h2>2. 部署方式与端口</h2>
|
||||
<table>
|
||||
<thead><tr><th><p>角色</p></th><th><p>端口</p></th><th><p>说明</p></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><p>P</p></td><td><p>30000 / 8998 / 9400-9403</p></td><td><p>HTTP / bootstrap / Mooncake client metrics</p></td></tr>
|
||||
<tr><td><p>D</p></td><td><p>30300</p></td><td><p>HTTP</p></td></tr>
|
||||
<tr><td><p>gateway</p></td><td><p>30400 / 29000</p></td><td><p>业务 / metrics</p></td></tr>
|
||||
<tr><td><p>mooncake-master</p></td><td><p>50051 / 18081 / 19003</p></td><td><p>store / metadata / metrics</p></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>默认入口:<code>http://127.0.0.1:30400</code>;当前 new-api/AgentX 直接使用该入口。</p>
|
||||
|
||||
<h2>3. 关键参数</h2>
|
||||
<p><b>P(prefill)</b>:tp4 / dp4 / ep4,megamoe + flashmla_mega,ep-redundant 32,DSpark block 3,decoder SWA bounded replay;chunked-prefill 131072 / max-prefill 131072,max-running 512;fastokens + tokenizer-worker 16;HiCache ratio 4、page_first_direct、direct、write_through、Mooncake L3;<b>cache 模式,无 buffer_only、无 optimistic</b>;mem-fraction 0.80。</p>
|
||||
<p><b>D(decode)</b>:tp4 / dp4 / ep4,ep-redundant 64,DSpark block 3,stream-interval 32,swa-prefix-tails 2048;max-running 512,cuda-graph-max-bs-decode 512;chunked 32768 / max-prefill 8192;fastokens + tokenizer-worker 16;mem-fraction 0.80。</p>
|
||||
<p><b>gateway</b>:镜像内 <code>sgl-model-gateway</code>;<b>无 P admission cap</b>(无 p-qps / p-queue-reqs-cap / p-queue-token-cap,td-admit 默认 off);保留 health check + circuit breaker;<code>--g9i-decode-max-running 512</code>。这是 c580 能跑通的关键。</p>
|
||||
<p><b>L3</b>:Mooncake master 4TB SSD offload;<code>offload_on_evict=false</code>;L3 配置见 <code>/data/ymk/ds41/mooncake-store-l3ssd.json</code>。</p>
|
||||
|
||||
<h2>4. 启动 / 重启流程</h2>
|
||||
<pre lang="bash"><code># 1) 停完整工作负载并确认 GPU 0
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 "nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
|
||||
|
||||
# 2) 清 L3 SSD(实验/基线前必须)
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
|
||||
# 3) 用完整平台 manifest 启动(不要只 kubectl patch gateway)
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml
|
||||
|
||||
# 4) 等 4 Pod 1/1,检查 P/D/gateway health 200,再 smoke
|
||||
curl -sS http://127.0.0.1:30400/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"/data/models/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"1+1=?"}],"max_tokens":8}'
|
||||
</code></pre>
|
||||
<p>启动基线:P <code>max_total_num_tokens≈16,158,208</code> / 128 per-rank / avail ≈23GB;D <code>58,859,520</code> / 128 per-rank / avail ≈79GB。正常 warmup 6,385 条约 6.5 分钟。</p>
|
||||
|
||||
<h2>5. 基线结果</h2>
|
||||
<table>
|
||||
<thead><tr><th><p>场景</p></th><th><p>RPS</p></th><th><p>TTFT p50/p95/p99</p></th><th><p>ITL p95</p></th><th><p>结论</p></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><p>c580 基线</p></td><td><p>24.045</p></td><td><p>1.743 / 3.772 / 6.580 s</p></td><td><p>10.74 ms</p></td><td><p>推荐工作点</p></td></tr>
|
||||
<tr><td><p>c620 首轮(清 L3)</p></td><td><p>25.998</p></td><td><p>1.814 / 4.041 / 8.257 s</p></td><td><p>11.69 ms</p></td><td><p>吞吐 +8%,尾部变差</p></td></tr>
|
||||
<tr><td><p>c620 不重启 r2/r3</p></td><td><p>25.85 / 25.61</p></td><td><p>p95 5.39 / 6.44 s,p99 9.56 / 10.79 s</p></td><td><p>10.96 / 10.68 ms</p></td><td><p>D KV 接近/打满,TTFT 尾部持续劣化,不建议连测</p></td></tr>
|
||||
<tr><td><p>buffer_only</p></td><td><p>23.749</p></td><td><p>2.153 / 4.904 / 10.483 s</p></td><td><p>10.32 ms</p></td><td><p>吞吐 -1.2%、TTFT 更差,不采用</p></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>结论:<b>c580 是稳定基线</b>;c620 只适合短时冲吞吐,且每轮之间必须重启/清池。buffer_only 不作为基线。</p>
|
||||
|
||||
<h2>6. 作为 Dynamo 底座</h2>
|
||||
<p>当前镜像是 <b>SGLang + Mooncake 0.3.14 + L3 SSD</b> 基线,<b>不包含 Dynamo/nixl</b>。后续套 Dynamo 需要以本镜像为 base 另装 Dynamo 1.6.x / nixl,旧 <code>gist-nvlink-7f6fe2244-dynamo-local-20261026</code> 是更早的 10-23 代码,不是本基线。</p>
|
||||
<p>Dynamo 叠加时保留本页的 P/D 参数、L3 SSD 和 gateway 入口;先做单机 4P4D 冒烟,再评估 Dynamo 的 P/D 角色、discovery 和 KV 传输配置。</p>
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"master_server_address": "127.0.0.1:50051",
|
||||
"metadata_server": "P2PHANDSHAKE",
|
||||
"global_segment_size": "200gb",
|
||||
"protocol": "tcp",
|
||||
"enable_ssd_offload": true,
|
||||
"ssd_offload_path": "/data/ymk/mooncake-ssd"
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
# L3 Mooncake SSD baseline requires the official 0.3.14 wheel.
|
||||
# Previous image shipped 0.3.13.post1; 0.3.14 contains the upstream SSD-offload fixes.
|
||||
# Wheel: https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
# sha256: 74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
|
||||
# Installed with --no-deps; the base image already provides the runtime deps.
|
||||
mooncake-transfer-engine-cuda13==0.3.14
|
||||
@@ -1,44 +0,0 @@
|
||||
# 2026-11-16 buffer_only + optimistic 首启记录
|
||||
|
||||
## 变更(相对 `stable-20261116-pre-optimistic`)
|
||||
|
||||
P:
|
||||
|
||||
- 增加 `--hicache-host-memory-mode buffer_only`(保留 `--hicache-write-policy write_through`、
|
||||
`--hicache-ratio 4`、mooncake L3)
|
||||
- 增加 `--optimistic-prefill-attempts 4`
|
||||
|
||||
D:
|
||||
|
||||
- 增加 `--tokenizer-worker-num 16`(与 P 相同)
|
||||
|
||||
其余角色/参数不变;网关保持 live 的 `p-queue-reqs-cap 64 / token-cap 32M / p-qps 40`。
|
||||
|
||||
## 启动前清理
|
||||
|
||||
- `kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s` → `Terminated 0/0`,GPU 0-7 全 0 MiB
|
||||
- `/data/ymk/mooncake-ssd`:2.3T → 4.0K;`/data` 7.5T available
|
||||
- `kmctl update cw ds41-gist-exp -f ds41-gist-exp-20261116-bufferonly-optimistic.yaml`
|
||||
|
||||
## 首启验证(2026-11-16 22:00 CST)
|
||||
|
||||
- 4 个 Pod 全 `1/1 Running`;P/D/gateway `/health` 全 200
|
||||
- P `server_args`:`hicache_host_memory_mode='buffer_only'`、`optimistic_prefill_attempts=4`;
|
||||
没有 "Optimistic prefill supports ..." 的禁用 warning
|
||||
- D `server_args`:`tokenizer_worker_num=16`;日志有 16 个 multi-tokenizer worker
|
||||
- 启动基线:P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
|
||||
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`
|
||||
- 冒烟:chat `/v1/chat/completions` 200,回答正确;16 并发 4k 字符请求全 200
|
||||
- **optimistic 已生效**:P 日志出现 `#optimistic-req: 1`(其余为 0)
|
||||
- P/D 日志无 Traceback / ERROR / assert
|
||||
|
||||
## 回滚
|
||||
|
||||
```bash
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/sglang/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
|
||||
```
|
||||
|
||||
代码仓库标签:`stable-20261116-pre-optimistic`。
|
||||
@@ -1,608 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T13:54:16Z"
|
||||
generation: 1
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30768532"
|
||||
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T14:01:52Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 1
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 1
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -1,68 +0,0 @@
|
||||
# 11/16 上线前稳定配置(ds41-gist-exp)
|
||||
|
||||
记录时间:2026-11-16 21:49 CST(13:49Z),在切换 `buffer_only + optimistic` 之前。
|
||||
|
||||
## 身份
|
||||
|
||||
- 平台 cw:`ds41-gist-exp` / `cw-dblft6sgrhagmpdt8ef0`
|
||||
- RBG:`ds41-gist-exp`(namespace `prj-d7te67bj0sq2fgdrmbig`)
|
||||
- 角色:`mooncake-master` / `p` / `d` / `gateway`
|
||||
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
|
||||
- P:GPU 0-3;D:GPU 4-7;全 hostNetwork / hostPID / privileged,钉 b300-01
|
||||
|
||||
## 文件
|
||||
|
||||
- `rbg-live-raw.yaml`:`kubectl get rbg -o yaml` 原始快照(含 status,仅存档)
|
||||
- `cw-ds41-gist-exp-11-16-pre-optimistic.yaml`:清理过平台字段的 RBG,可直接用于回滚更新
|
||||
- `cw-platform.yaml`:`kmctl get cw -o yaml` 的平台清单(`Spec.Manifest` 是旧值,见下方漂移)
|
||||
- 本说明
|
||||
|
||||
## 生效配置摘要
|
||||
|
||||
P:
|
||||
|
||||
- `tp4 / dp4 / ep4`、`megamoe`、`flashmla_mega`、`ep-redundant 32`
|
||||
- DSPARK block 3、decoder SWA bounded replay、`swa-full-tokens-ratio 0.12`
|
||||
- `chunked-prefill-size 131072`、`max-prefill-tokens 131072`
|
||||
- `max-running-requests 512`、`tokenizer-worker-num 16`
|
||||
- `hicache-ratio 4`、`hicache-mem-layout page_first_direct`、`hicache-io-backend direct`
|
||||
- `hicache-write-policy write_through`、`hicache-storage-backend mooncake`
|
||||
- **默认 host memory mode(cache),没有 `--hicache-host-memory-mode buffer_only`**
|
||||
- **没有 `--optimistic-prefill-attempts`**(`#optimistic-req` 为 0)
|
||||
- L3 SSD:4 TB `mooncake-ssd`,master `--offload_on_evict=false`
|
||||
|
||||
D:
|
||||
|
||||
- 同并行 tp4/dp4/ep4、`ep-redundant 64`、DSPARK block 3
|
||||
- `max-running-requests 512`、`cuda-graph-max-bs-decode 512`
|
||||
- `chunked-prefill-size 32768`、`max-prefill-tokens 8192`
|
||||
- **没有 `--tokenizer-worker-num`**(默认 1)
|
||||
- `mem-fraction-static 0.80`
|
||||
|
||||
Gateway:
|
||||
|
||||
- 实际 live:`--g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256`
|
||||
- `--g9i-regular-p-queue-reject-secs 0`、`--g9i-decode-max-running 512`
|
||||
- 网关二进制 hostPath:`/data/ymk/gw-ns-est-td/sgl-model-gateway`
|
||||
|
||||
> ⚠️ 平台 `cw-platform.yaml` 的 `Spec.Manifest` 里 gateway 仍是旧的
|
||||
> `p-queue-reqs-cap 32 / token-cap 16000000 / p-qps 20`;**live RBG 是 64 / 32M / 40**。
|
||||
> 本次快照与回滚文件都以 live RBG 为准,避免回滚时把网关参数打回旧值。
|
||||
|
||||
## 切换前状态
|
||||
|
||||
- 所有 Pod `1/1 Running`;P/D/gateway `/health` 200。
|
||||
- 为切 `buffer_only` 做 L3 SSD 清理前,`/data/ymk/mooncake-ssd` 占用 **2.3T**;清理后 4.0K,`/data` 7.5T available。
|
||||
- 启动基线(切换后复核):P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
|
||||
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`。
|
||||
|
||||
## 回滚
|
||||
|
||||
```bash
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
|
||||
```
|
||||
|
||||
回滚前确认 `kmctl get cw ds41-gist-exp` 为 `Terminated 0/0`、GPU 0-7 为 0 MiB。
|
||||
-488
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
File diff suppressed because one or more lines are too long
@@ -1,608 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T00:16:46Z"
|
||||
generation: 9
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30735810"
|
||||
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T12:36:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 9
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 9
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -1,608 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T00:16:46Z"
|
||||
generation: 9
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30735810"
|
||||
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T12:36:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 9
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 9
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -1,13 +0,0 @@
|
||||
# 2026-11-17 cache + gateway no-limit 实验参数(失败记录)
|
||||
|
||||
- P: hicache_host_memory_mode=cache, optimistic_prefill_attempts=0, tokenizer-worker-num=16,
|
||||
hicache ratio 4 / write_through / L3 mooncake, L3 SSD 4T.
|
||||
- D: tokenizer-worker-num=16.
|
||||
- Gateway: g9i-td-admit off, p-queue-reqs-cap 0, p-queue-token-cap 0, p-qps 0,
|
||||
decode-max-running 1000000, disable-circuit-breaker, disable-health-check.
|
||||
- Image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
- c580 run c580-cache-nolimit-20261117: warmup 4,720/6,385 (74%), 02:33:32 失败
|
||||
(ServerDisconnected/warmup_failure;P/D/gateway 0 restart,无 503)。
|
||||
- 观察:P 单批 32k/23k chunk 正常,但每 rank 同时仅 1 个请求;gateway
|
||||
smg_worker_requests_active 峰值 1,840,P queue 峰值 503,gateway 约 1 核。
|
||||
- 结论:该 no-limit/no-cb 网关配置不适合作为 c580 基线;仅作异常记录。
|
||||
@@ -1,488 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -1,608 +0,0 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T13:54:16Z"
|
||||
generation: 6
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "31040141"
|
||||
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 1000000 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-17T01:58:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 6
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 6
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -443,15 +443,23 @@ def late_layer_tail_lens(
|
||||
extend_lens_cpu: list[int],
|
||||
tail_len: int,
|
||||
whole_extend: Optional[list[bool]] = None,
|
||||
prompt_rest_lens: Optional[list[int]] = None,
|
||||
) -> list[int]:
|
||||
"""Each request's late-layer tail: its last min(tail_len, extend_len) tokens,
|
||||
or its whole extend where whole_extend is set."""
|
||||
if whole_extend is None:
|
||||
return [min(tail_len, n) for n in extend_lens_cpu]
|
||||
return [
|
||||
n if whole else min(tail_len, n)
|
||||
for n, whole in zip(extend_lens_cpu, whole_extend)
|
||||
]
|
||||
or its whole extend where whole_extend is set. With prompt_rest_lens (prompt
|
||||
tokens left after each extend), only its rows in the prompt's last tail_len
|
||||
tokens, and at least its last row."""
|
||||
if whole_extend is not None and any(whole_extend):
|
||||
return [
|
||||
n if whole else min(tail_len, n)
|
||||
for n, whole in zip(extend_lens_cpu, whole_extend)
|
||||
]
|
||||
if prompt_rest_lens is not None:
|
||||
return [
|
||||
min(n, max(1, tail_len - rest))
|
||||
for n, rest in zip(extend_lens_cpu, prompt_rest_lens, strict=True)
|
||||
]
|
||||
return [min(tail_len, n) for n in extend_lens_cpu]
|
||||
|
||||
|
||||
def late_layer_tail_layout(
|
||||
@@ -461,13 +469,15 @@ def late_layer_tail_layout(
|
||||
tail_len: int,
|
||||
device: torch.device,
|
||||
whole_extend: Optional[list[bool]] = None,
|
||||
prompt_rest_lens: Optional[list[int]] = None,
|
||||
) -> tuple[torch.Tensor, list[int], torch.Tensor]:
|
||||
"""Tail rows of each prefill extend: its last min(tail_len, extend_len) tokens,
|
||||
or its whole extend where whole_extend is set.
|
||||
Returns (token indices into the extend, per-request tail lengths, per-row
|
||||
absolute window floor). The floor is seq_len - tail_len, so a tail shorter
|
||||
than tail_len reads rows the request wrote before this extend. A whole
|
||||
extend keeps every row's plain window."""
|
||||
extend keeps every row's plain window. With prompt_rest_lens, tails and
|
||||
floors follow the prompt's end instead (late_layer_tail_lens, _tail_floors)."""
|
||||
if whole_extend is not None and any(whole_extend):
|
||||
return _whole_extend_tail_layout(
|
||||
extend_lens_cpu=extend_lens_cpu,
|
||||
@@ -476,16 +486,19 @@ def late_layer_tail_layout(
|
||||
device=device,
|
||||
whole_extend=whole_extend,
|
||||
)
|
||||
tail_lens_cpu = late_layer_tail_lens(extend_lens_cpu, tail_len)
|
||||
tail_lens_cpu = late_layer_tail_lens(
|
||||
extend_lens_cpu, tail_len, prompt_rest_lens=prompt_rest_lens
|
||||
)
|
||||
floors_cpu = _tail_floors(seq_lens_cpu, tail_len, prompt_rest_lens)
|
||||
if len(extend_lens_cpu) == 1:
|
||||
n, t, s = extend_lens_cpu[0], tail_lens_cpu[0], seq_lens_cpu[0]
|
||||
floor = torch.full((t,), max(0, s - tail_len), dtype=torch.int32, device=device)
|
||||
n, t = extend_lens_cpu[0], tail_lens_cpu[0]
|
||||
floor = torch.full((t,), floors_cpu[0], dtype=torch.int32, device=device)
|
||||
return torch.arange(n - t, n, device=device), tail_lens_cpu, floor
|
||||
# One H2D copy for the three length vectors; launch count does not grow with bs.
|
||||
lens = async_h2d(
|
||||
[extend_lens_cpu, tail_lens_cpu, seq_lens_cpu], dtype=torch.int64, device=device
|
||||
[extend_lens_cpu, tail_lens_cpu, floors_cpu], dtype=torch.int64, device=device
|
||||
)
|
||||
extend_lens, tail_lens, seq_lens = lens[0], lens[1], lens[2]
|
||||
extend_lens, tail_lens, floors = lens[0], lens[1], lens[2]
|
||||
total = sum(tail_lens_cpu)
|
||||
req = torch.repeat_interleave(
|
||||
torch.arange(len(tail_lens_cpu), device=device), tail_lens, output_size=total
|
||||
@@ -495,10 +508,23 @@ def late_layer_tail_layout(
|
||||
- (torch.cumsum(tail_lens, 0) - tail_lens)[req]
|
||||
)
|
||||
token_indices = (torch.cumsum(extend_lens, 0) - tail_lens)[req] + offs
|
||||
floor = (seq_lens - tail_len).clamp_min(0)[req].to(torch.int32)
|
||||
floor = floors[req].to(torch.int32)
|
||||
return token_indices, tail_lens_cpu, floor
|
||||
|
||||
|
||||
def _tail_floors(
|
||||
seq_lens_cpu: list[int], tail_len: int, prompt_rest_lens: Optional[list[int]]
|
||||
) -> list[int]:
|
||||
"""Each request's window floor: tail_len below its extend's end, or below its
|
||||
prompt's end, clamped to its last row, whose own KV a lone tail row reads."""
|
||||
if prompt_rest_lens is None:
|
||||
return [max(0, s - tail_len) for s in seq_lens_cpu]
|
||||
return [
|
||||
min(max(0, s + rest - tail_len), max(0, s - 1))
|
||||
for s, rest in zip(seq_lens_cpu, prompt_rest_lens, strict=True)
|
||||
]
|
||||
|
||||
|
||||
def _whole_extend_tail_layout(
|
||||
*,
|
||||
extend_lens_cpu: list[int],
|
||||
|
||||
@@ -98,7 +98,6 @@ class Parallel(msgspec.Struct):
|
||||
"follow_bootstrap_room",
|
||||
"total_requests",
|
||||
"total_tokens",
|
||||
"uncached_tokens",
|
||||
],
|
||||
),
|
||||
] = "auto"
|
||||
|
||||
@@ -728,6 +728,7 @@ class TboForwardBatchPreparer:
|
||||
"extend_prefix_lens_cpu",
|
||||
"extend_seq_lens_cpu",
|
||||
"extend_logprob_start_lens_cpu",
|
||||
"extend_prompt_rest_lens_cpu",
|
||||
"lora_ids",
|
||||
"rids",
|
||||
]:
|
||||
|
||||
@@ -51,7 +51,6 @@ from sglang.srt.disaggregation.utils import (
|
||||
)
|
||||
from sglang.srt.environ import envs
|
||||
from sglang.srt.mem_cache.page_interleave import get_kv_shard_group
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.trace import TraceNullContext, TraceReqContext
|
||||
from sglang.srt.runtime_context import (
|
||||
get_device,
|
||||
@@ -2092,10 +2091,6 @@ class CommonKVSender(BaseKVSender):
|
||||
self._transfer_num_state_indices = 0
|
||||
self._transfer_start_time: Optional[float] = None
|
||||
self._early_send_wait_event: Optional[object] = None
|
||||
# Detail tracing: chunk classification only. Enqueue timestamps,
|
||||
# chunk_seq and the per-request first-enqueue are assigned by the
|
||||
# manager at the real queue.put() point, after all early returns.
|
||||
self._send_kind = ""
|
||||
self.trace_ctx = TraceNullContext()
|
||||
if self.kv_mgr.sender_trace_module and get_observability().enable_trace:
|
||||
self.trace_ctx = TraceReqContext(
|
||||
@@ -2294,19 +2289,6 @@ class CommonKVSender(BaseKVSender):
|
||||
):
|
||||
self._transfer_start_time = time.perf_counter()
|
||||
|
||||
obs = None
|
||||
if batch_telemetry.mooncake_detail_enabled():
|
||||
# Metadata only; the manager stamps chunk_seq and enqueue times
|
||||
# when it is certain the chunk is actually put on the queue.
|
||||
obs = {
|
||||
"loop_seq": int(batch_telemetry.current_loop_seq()),
|
||||
"send_kind": str(self._send_kind or ""),
|
||||
"num_pages": int(len(kv_indices)),
|
||||
"approx_bytes": int(
|
||||
len(kv_indices) * getattr(self.kv_mgr, "kv_item_lens_sum", 0)
|
||||
),
|
||||
}
|
||||
|
||||
# The early-send event belongs to exactly one chunk; the Mooncake and
|
||||
# Mori workers synchronize on it before reading source KV.
|
||||
wait_event = self._early_send_wait_event
|
||||
@@ -2322,27 +2304,9 @@ class CommonKVSender(BaseKVSender):
|
||||
wait_event=wait_event,
|
||||
indices_event=indices_event,
|
||||
trace_ctx=self.trace_ctx.copy_for_thread(),
|
||||
obs=obs,
|
||||
)
|
||||
self._record_transfer_indices(kv_indices, state_indices)
|
||||
|
||||
def pop_chunk_join_stats(self) -> Optional[dict]:
|
||||
"""Take (and free) this room's chunk join summary."""
|
||||
state = getattr(self.kv_mgr, "_room_detail", None)
|
||||
if state is None:
|
||||
return None
|
||||
return state.pop_join(self.bootstrap_room)
|
||||
|
||||
def reset_chunk_detail_state(self) -> None:
|
||||
"""Drop per-room detail state and mark it cancelled (abort / retract).
|
||||
|
||||
The mark is what prevents a worker attempt that already passed its
|
||||
status check from recreating stats after this cleanup.
|
||||
"""
|
||||
state = getattr(self.kv_mgr, "_room_detail", None)
|
||||
if state is not None:
|
||||
state.reset(self.bootstrap_room)
|
||||
|
||||
def poll(self) -> KVPoll:
|
||||
return self._poll_with_status(self.kv_mgr.check_status)
|
||||
|
||||
@@ -2461,9 +2425,6 @@ class CommonKVSender(BaseKVSender):
|
||||
def abort(self):
|
||||
if self.kv_mgr.deferred_bootstrap is not None and not self._owns_bootstrap_room:
|
||||
return
|
||||
# Drop per-room detail bookkeeping: an aborted request never reaches
|
||||
# the poll path that normally consumes it (review finding).
|
||||
self.reset_chunk_detail_state()
|
||||
self.kv_mgr.record_failure(
|
||||
self.bootstrap_room,
|
||||
"Aborted by AbortReq.",
|
||||
|
||||
@@ -36,24 +36,6 @@ class TransferKVChunk:
|
||||
# Must complete before reading asynchronously copied host indices.
|
||||
indices_event: Optional[object] = None
|
||||
|
||||
# --- Optional observation fields (SGLANG_TRACE_MOONCAKE_DETAIL=1) ------
|
||||
# Filled at the real enqueue point (add_transfer_request), never at the
|
||||
# send_kv_chunk entry, so skipped paths cannot create holes in chunk_seq.
|
||||
chunk_seq: int = 0 # per-sender monotonic enqueue index
|
||||
loop_seq: int = 0 # scheduler loop that produced this chunk
|
||||
send_kind: str = "" # cached_prefix / middle / last
|
||||
enqueue_mono_ns: int = 0 # first enqueue of this chunk, monotonic clock
|
||||
enqueue_wall_ns: int = 0 # first enqueue of this chunk, wall clock
|
||||
last_enqueue_mono_ns: int = 0 # updated on every (re-)enqueue
|
||||
last_enqueue_wall_ns: int = 0 # wall clock of the last (re-)enqueue
|
||||
dequeue_mono_ns: int = 0 # current attempt dequeue; 0 outside the worker
|
||||
queue_wait_accum_ns: int = 0 # completed queue waits from earlier attempts
|
||||
queue_id: int = -1
|
||||
requeue_count: int = 0
|
||||
num_pages: int = 0
|
||||
approx_bytes: int = 0
|
||||
observe: bool = False # True when detail tracing populated this chunk
|
||||
|
||||
|
||||
def pack_list_of_buffers(buffers: List[bytes]) -> bytes:
|
||||
if not buffers:
|
||||
|
||||
@@ -60,7 +60,6 @@ from sglang.srt.disaggregation.utils import (
|
||||
)
|
||||
from sglang.srt.distributed.parallel_state import get_mooncake_transfer_engine
|
||||
from sglang.srt.environ import envs
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.mooncake_trace import (
|
||||
MooncakeRequestStage,
|
||||
mooncake_trace_func,
|
||||
@@ -357,28 +356,6 @@ def _uses_flat_entry_layout(
|
||||
return mla_backend or force_flat or has_layer_ids or is_single_region_swa
|
||||
|
||||
|
||||
def materialize_state_indices(state_indices, indices_event):
|
||||
"""Convert pinned state tensors to numpy after the event has completed.
|
||||
|
||||
``indices_event`` must already be synchronized by the caller. A tensor
|
||||
without an event is a configuration/implementation error: reading it would
|
||||
race the scheduler's non-blocking D2H copy.
|
||||
"""
|
||||
if not state_indices:
|
||||
return state_indices
|
||||
import torch
|
||||
|
||||
if not any(isinstance(t, torch.Tensor) for t in state_indices):
|
||||
return state_indices
|
||||
if indices_event is None:
|
||||
raise RuntimeError(
|
||||
"async state indices require an indices_event before materialization"
|
||||
)
|
||||
return [
|
||||
t.numpy() if isinstance(t, torch.Tensor) else t for t in state_indices
|
||||
]
|
||||
|
||||
|
||||
class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
sender_trace_module = "mooncake"
|
||||
|
||||
@@ -409,10 +386,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
# transfer blocks over zmq when the intra-node NVLink transport is
|
||||
# active (it cannot reach host memory). Populated on bootstrap.
|
||||
self._session_endpoint_map = {}
|
||||
# Detail tracing (SGLANG_TRACE_MOONCAKE_DETAIL=1; empty otherwise).
|
||||
# Lock-protected per-room state: abort/retract and the worker's join
|
||||
# update must not interleave into resurrected stats.
|
||||
self._room_detail = batch_telemetry.RoomDetailState()
|
||||
self.enable_staging = envs.SGLANG_DISAGG_STAGING_BUFFER.get()
|
||||
self.max_transfer_batch_indices = (
|
||||
envs.SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES.get()
|
||||
@@ -789,10 +762,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
# block other rooms on this single worker thread.
|
||||
with self._staging_ctx.watermark_cv:
|
||||
self._staging_ctx.watermark_cv.wait(STAGING_WATERMARK_WAIT_S)
|
||||
if getattr(kv_chunk, "observe", False):
|
||||
batch_telemetry.note_chunk_requeue(
|
||||
kv_chunk, time.monotonic_ns(), time.time_ns()
|
||||
)
|
||||
queue.put(kv_chunk)
|
||||
return (-1, True)
|
||||
|
||||
@@ -2747,65 +2716,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
if self.enable_deferred_decode_kv_release:
|
||||
self.poison_deferred_ack_room(room)
|
||||
|
||||
def _emit_chunk_attempt(
|
||||
self,
|
||||
kv_chunk: TransferKVChunk,
|
||||
*,
|
||||
status: str,
|
||||
queue_wait_ns: int,
|
||||
queue_wait_total_ns: int,
|
||||
source_wait_ns: int,
|
||||
indices_wait_ns: int,
|
||||
kv_transfer_ns: int,
|
||||
state_transfer_ns: int,
|
||||
aux_transfer_ns: int,
|
||||
work_start_ns: int,
|
||||
dequeue_wall_ns: int,
|
||||
n_destinations: int,
|
||||
source_wait_wall=None,
|
||||
indices_wait_wall=None,
|
||||
) -> None:
|
||||
"""Close one worker attempt: chunk span + per-room join summary."""
|
||||
now_mono_ns = time.monotonic_ns()
|
||||
now_wall_ns = time.time_ns()
|
||||
attempt_wall_ns = max(0, now_mono_ns - int(work_start_ns))
|
||||
worker_busy_ns = max(
|
||||
0, attempt_wall_ns - int(source_wait_ns) - int(indices_wait_ns)
|
||||
)
|
||||
batch_telemetry.emit_mooncake_chunk(
|
||||
chunk=kv_chunk,
|
||||
status=status,
|
||||
queue_wait_ns=queue_wait_ns,
|
||||
queue_wait_total_ns=queue_wait_total_ns,
|
||||
source_wait_ns=source_wait_ns,
|
||||
indices_wait_ns=indices_wait_ns,
|
||||
kv_transfer_ns=kv_transfer_ns,
|
||||
state_transfer_ns=state_transfer_ns,
|
||||
aux_transfer_ns=aux_transfer_ns,
|
||||
attempt_wall_ns=attempt_wall_ns,
|
||||
dequeue_wall_ns=dequeue_wall_ns,
|
||||
end_wall_ns=now_wall_ns,
|
||||
n_destinations=n_destinations,
|
||||
source_wait_wall=source_wait_wall,
|
||||
indices_wait_wall=indices_wait_wall,
|
||||
dp_rank=self.attn_dp_rank,
|
||||
tp_rank=self.attn_tp_rank,
|
||||
)
|
||||
# The status was checked before this call; the registry re-checks the
|
||||
# cancellation mark under its lock, which is what makes an abort that
|
||||
# lands in between unable to resurrect stats (review finding).
|
||||
self._room_detail.record_join(
|
||||
kv_chunk.room,
|
||||
status=status,
|
||||
room_failed=self.check_status(kv_chunk.room) == KVPoll.Failed,
|
||||
now_wall_ns=now_wall_ns,
|
||||
queue_wait_total_ns=queue_wait_total_ns,
|
||||
worker_busy_ns=worker_busy_ns,
|
||||
enqueue_wall_ns=kv_chunk.enqueue_wall_ns,
|
||||
is_last_chunk=kv_chunk.is_last_chunk,
|
||||
chunk_seq=kv_chunk.chunk_seq,
|
||||
)
|
||||
|
||||
def transfer_worker(
|
||||
self,
|
||||
queue: FastQueue,
|
||||
@@ -2830,34 +2740,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
# _stopped alone can never wake a parked worker during a role switch.
|
||||
if kv_chunk is None:
|
||||
break
|
||||
_obs = bool(getattr(kv_chunk, "observe", False))
|
||||
_dequeue_mono_ns = time.monotonic_ns() if _obs else 0
|
||||
_dequeue_wall_ns = time.time_ns() if _obs else 0
|
||||
if _obs:
|
||||
# Needed by note_chunk_requeue(): queue wait ends here and
|
||||
# must not include worker/watermark time.
|
||||
kv_chunk.dequeue_mono_ns = _dequeue_mono_ns
|
||||
_queue_wait_ns = 0
|
||||
_source_wait_ns = 0
|
||||
_source_wait_wall = None
|
||||
_indices_wait_ns = 0
|
||||
_indices_wait_wall = None
|
||||
_kv_transfer_ns = 0
|
||||
_state_transfer_ns = 0
|
||||
_aux_transfer_ns = 0
|
||||
_n_dest = 0
|
||||
if _obs:
|
||||
# queue wait is not part of worker busy time (design §5.2).
|
||||
# Keep the attempt window (child span) separate from the
|
||||
# cumulative value across deferred attempts (review finding).
|
||||
(
|
||||
_attempt_queue_wait_ns,
|
||||
_queue_wait_ns,
|
||||
) = batch_telemetry.split_queue_wait(
|
||||
kv_chunk.queue_wait_accum_ns,
|
||||
kv_chunk.last_enqueue_mono_ns,
|
||||
_dequeue_mono_ns,
|
||||
)
|
||||
if self.enable_trace:
|
||||
kv_chunk.trace_ctx.rebuild_thread_context()
|
||||
kv_chunk.trace_ctx.trace_slice_start(
|
||||
@@ -2883,32 +2765,13 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
kv_chunk.room in self.request_status
|
||||
and self.check_status(kv_chunk.room) != KVPoll.Failed
|
||||
):
|
||||
_wait_t0 = time.monotonic_ns() if _obs else 0
|
||||
_wait_wall0 = time.time_ns() if _obs else 0
|
||||
kv_chunk.wait_event.synchronize()
|
||||
if _wait_t0:
|
||||
_source_wait_ns += time.monotonic_ns() - _wait_t0
|
||||
_wait_wall1 = time.time_ns()
|
||||
_source_wait_wall = (_wait_wall0, _wait_wall1)
|
||||
kv_chunk.wait_event = None
|
||||
_ind_t0 = time.monotonic_ns() if _obs else 0
|
||||
_ind_wall0 = time.time_ns() if _obs else 0
|
||||
indices_event = kv_chunk.indices_event
|
||||
if indices_event is not None:
|
||||
indices_event.synchronize()
|
||||
if kv_chunk.indices_event is not None:
|
||||
kv_chunk.indices_event.synchronize()
|
||||
kv_chunk.indices_event = None
|
||||
if isinstance(kv_chunk.prefill_kv_indices, torch.Tensor):
|
||||
kv_chunk.prefill_kv_indices = kv_chunk.prefill_kv_indices.numpy()
|
||||
# First async-state version: the pinned SWA state tensor is
|
||||
# materialized only after the event above. Raises if a tensor
|
||||
# arrives without one (would race the scheduler's D2H).
|
||||
kv_chunk.state_indices = materialize_state_indices(
|
||||
kv_chunk.state_indices, indices_event
|
||||
)
|
||||
if _ind_t0:
|
||||
# synchronize + host indices availability, per design §5.2.
|
||||
_indices_wait_ns += time.monotonic_ns() - _ind_t0
|
||||
_indices_wait_wall = (_ind_wall0, time.time_ns())
|
||||
|
||||
if (
|
||||
kv_chunk.room not in self.request_status
|
||||
@@ -2924,23 +2787,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
thread_finish_flag=True,
|
||||
)
|
||||
self._staging_outstanding.pop(kv_chunk.room, None)
|
||||
if _obs:
|
||||
self._emit_chunk_attempt(
|
||||
kv_chunk,
|
||||
status="skipped",
|
||||
queue_wait_ns=_attempt_queue_wait_ns,
|
||||
queue_wait_total_ns=_queue_wait_ns,
|
||||
source_wait_ns=_source_wait_ns,
|
||||
indices_wait_ns=_indices_wait_ns,
|
||||
kv_transfer_ns=_kv_transfer_ns,
|
||||
state_transfer_ns=_state_transfer_ns,
|
||||
aux_transfer_ns=_aux_transfer_ns,
|
||||
work_start_ns=_dequeue_mono_ns,
|
||||
dequeue_wall_ns=_dequeue_wall_ns,
|
||||
n_destinations=_n_dest,
|
||||
source_wait_wall=_source_wait_wall,
|
||||
indices_wait_wall=_indices_wait_wall,
|
||||
)
|
||||
if self.enable_deferred_decode_kv_release:
|
||||
# Skipped => nothing written for this aborted room; ack.
|
||||
self._maybe_ack_drained_abort(kv_chunk.room)
|
||||
@@ -3036,7 +2882,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
skip_kv, skip_state = self._get_dsa_cache_transfer_skip_flags(
|
||||
target_rank_registration_info
|
||||
)
|
||||
_kv_t0 = time.monotonic_ns() if _obs else 0
|
||||
if (
|
||||
len(kv_chunk.prefill_kv_indices) == 0
|
||||
or not self.kv_args.kv_data_ptrs
|
||||
@@ -3109,9 +2954,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
)
|
||||
if deferred:
|
||||
staging_deferred = True
|
||||
if _kv_t0:
|
||||
_kv_transfer_ns += time.monotonic_ns() - _kv_t0
|
||||
_n_dest += 1
|
||||
# Chunk re-enqueued; stop processing remaining reqs for this chunk
|
||||
break
|
||||
else:
|
||||
@@ -3126,9 +2968,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
executor,
|
||||
target_rank_registration_info.dst_kv_layer_ids,
|
||||
)
|
||||
if _kv_t0:
|
||||
_kv_transfer_ns += time.monotonic_ns() - _kv_t0
|
||||
_n_dest += 1
|
||||
if ret != 0:
|
||||
with self.session_lock:
|
||||
self.session_failures[req.mooncake_session_id] += 1
|
||||
@@ -3150,15 +2989,12 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
|
||||
if kv_chunk.is_last_chunk:
|
||||
if kv_chunk.state_indices and not skip_state:
|
||||
_state_t0 = time.monotonic_ns() if _obs else 0
|
||||
state_rc = self.maybe_send_extra(
|
||||
req,
|
||||
kv_chunk.state_indices,
|
||||
executor,
|
||||
target_rank_registration_info,
|
||||
)
|
||||
if _state_t0:
|
||||
_state_transfer_ns += time.monotonic_ns() - _state_t0
|
||||
# Before the Failed check: an ABORT during the
|
||||
# send also marks the room Failed.
|
||||
if state_rc != 0:
|
||||
@@ -3189,14 +3025,11 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
break
|
||||
|
||||
# Only the last chunk we need to send the aux data
|
||||
_aux_t0 = time.monotonic_ns() if _obs else 0
|
||||
ret = self.send_aux(
|
||||
req,
|
||||
kv_chunk.prefill_aux_index,
|
||||
target_rank_registration_info.dst_aux_ptrs,
|
||||
)
|
||||
if _aux_t0:
|
||||
_aux_transfer_ns += time.monotonic_ns() - _aux_t0
|
||||
if ret != 0:
|
||||
# The aux write targets decode's metadata
|
||||
# slot, which the deferred release also frees.
|
||||
@@ -3238,45 +3071,8 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
)
|
||||
|
||||
if staging_deferred:
|
||||
if _obs:
|
||||
self._emit_chunk_attempt(
|
||||
kv_chunk,
|
||||
status="deferred",
|
||||
queue_wait_ns=_attempt_queue_wait_ns,
|
||||
queue_wait_total_ns=_queue_wait_ns,
|
||||
source_wait_ns=_source_wait_ns,
|
||||
indices_wait_ns=_indices_wait_ns,
|
||||
kv_transfer_ns=_kv_transfer_ns,
|
||||
state_transfer_ns=_state_transfer_ns,
|
||||
aux_transfer_ns=_aux_transfer_ns,
|
||||
work_start_ns=_dequeue_mono_ns,
|
||||
dequeue_wall_ns=_dequeue_wall_ns,
|
||||
n_destinations=_n_dest,
|
||||
source_wait_wall=_source_wait_wall,
|
||||
indices_wait_wall=_indices_wait_wall,
|
||||
)
|
||||
continue
|
||||
|
||||
if _obs:
|
||||
self._emit_chunk_attempt(
|
||||
kv_chunk,
|
||||
status="ok"
|
||||
if self.check_status(kv_chunk.room) != KVPoll.Failed
|
||||
else "failed",
|
||||
queue_wait_ns=_attempt_queue_wait_ns,
|
||||
queue_wait_total_ns=_queue_wait_ns,
|
||||
source_wait_ns=_source_wait_ns,
|
||||
indices_wait_ns=_indices_wait_ns,
|
||||
kv_transfer_ns=_kv_transfer_ns,
|
||||
state_transfer_ns=_state_transfer_ns,
|
||||
aux_transfer_ns=_aux_transfer_ns,
|
||||
work_start_ns=_dequeue_mono_ns,
|
||||
dequeue_wall_ns=_dequeue_wall_ns,
|
||||
n_destinations=_n_dest,
|
||||
source_wait_wall=_source_wait_wall,
|
||||
indices_wait_wall=_indices_wait_wall,
|
||||
)
|
||||
|
||||
self._staging_outstanding[kv_chunk.room] -= 1
|
||||
if self.enable_deferred_decode_kv_release:
|
||||
# In-flight write finished; if aborted and nothing outstanding,
|
||||
@@ -3300,15 +3096,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
self._session_endpoint_map.pop(sid, None)
|
||||
self.transfer_infos.pop(kv_chunk.room)
|
||||
self.req_to_decode_prefix_len.pop(kv_chunk.room, None)
|
||||
# Success: the scheduler poll still needs the join summary.
|
||||
# Failed/absent: nothing will consume it, drop it here.
|
||||
self._room_detail.teardown(
|
||||
kv_chunk.room,
|
||||
keep_join=(
|
||||
self.request_status.get(kv_chunk.room)
|
||||
== KVPoll.Success
|
||||
),
|
||||
)
|
||||
if self.enable_staging:
|
||||
# Purge prefetch bookkeeping for the finished room.
|
||||
# Snapshot first: the scheduler thread adds concurrently.
|
||||
@@ -3545,7 +3332,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
wait_event: Optional[object] = None,
|
||||
indices_event: Optional[object] = None,
|
||||
trace_ctx: Optional[Union[TraceReqContext, TraceNullContext]] = None,
|
||||
obs: Optional[dict] = None,
|
||||
):
|
||||
assert self.disaggregation_mode == DisaggregationMode.PREFILL
|
||||
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
|
||||
@@ -3575,23 +3361,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
if trace_ctx is None:
|
||||
trace_ctx = TraceNullContext()
|
||||
|
||||
if obs is not None:
|
||||
# Stamp only now: every earlier return (failed room, dummy rank,
|
||||
# zero-page skip) must not consume a chunk_seq or a first-enqueue
|
||||
# timestamp (review finding).
|
||||
now_mono_ns = time.monotonic_ns()
|
||||
now_wall_ns = time.time_ns()
|
||||
chunk_seq, _first_wall_ns = self._room_detail.begin_chunk(
|
||||
bootstrap_room, now_mono_ns, now_wall_ns
|
||||
)
|
||||
obs = dict(obs)
|
||||
obs.update(
|
||||
chunk_seq=chunk_seq,
|
||||
enqueue_mono_ns=now_mono_ns,
|
||||
enqueue_wall_ns=now_wall_ns,
|
||||
last_enqueue_wall_ns=now_wall_ns,
|
||||
)
|
||||
|
||||
self.transfer_queues[shard_idx].put(
|
||||
TransferKVChunk(
|
||||
room=bootstrap_room,
|
||||
@@ -3604,10 +3373,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
trace_ctx=trace_ctx,
|
||||
wait_event=wait_event,
|
||||
indices_event=indices_event,
|
||||
**(obs or {}),
|
||||
queue_id=int(shard_idx),
|
||||
observe=obs is not None,
|
||||
last_enqueue_mono_ns=int((obs or {}).get("enqueue_mono_ns", 0)),
|
||||
)
|
||||
)
|
||||
|
||||
@@ -3786,10 +3551,6 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
|
||||
|
||||
class MooncakeKVSender(CommonKVSender):
|
||||
supports_host_tensor_indices = True
|
||||
# First async state-payload version (SWA-only): the scheduler copies the
|
||||
# SWA state indices to a pinned buffer and the worker materializes them
|
||||
# after kv_chunk.indices_event.
|
||||
supports_async_state_indices = True
|
||||
|
||||
@mooncake_trace_func(MooncakeRequestStage.MOONCAKE_SEND)
|
||||
def send(
|
||||
|
||||
@@ -804,7 +804,6 @@ class MoriKVManager(CommonKVManager):
|
||||
wait_event: Optional[object] = None,
|
||||
indices_event: Optional[object] = None,
|
||||
trace_ctx=None,
|
||||
obs: Optional[dict] = None,
|
||||
) -> None:
|
||||
assert self.disaggregation_mode == DisaggregationMode.PREFILL
|
||||
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
|
||||
|
||||
@@ -3267,7 +3267,6 @@ class NixlKVManager(StagingManagerMixin, CommonKVManager):
|
||||
wait_event=None,
|
||||
indices_event=None,
|
||||
trace_ctx=None,
|
||||
obs: Optional[dict] = None,
|
||||
):
|
||||
assert self.disaggregation_mode == DisaggregationMode.PREFILL
|
||||
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
|
||||
|
||||
@@ -77,7 +77,6 @@ from sglang.srt.mem_cache.base_prefix_cache import CacheRequestOutcome
|
||||
from sglang.srt.mem_cache.common import (
|
||||
checkpoint_kv_cache,
|
||||
kv_to_page_indices,
|
||||
kv_to_page_indices_pinned,
|
||||
kv_to_page_num,
|
||||
release_kv_cache,
|
||||
suffix_page_indices,
|
||||
@@ -85,7 +84,6 @@ from sglang.srt.mem_cache.common import (
|
||||
from sglang.srt.mem_cache.deepseek_v4_memory_pool import DeepSeekV4TokenToKVPool
|
||||
from sglang.srt.mem_cache.memory_pool import HybridLinearKVPool
|
||||
from sglang.srt.mem_cache.unified_radix_cache import UnifiedRadixCache
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.req_time_stats import set_schedule_time_batch
|
||||
from sglang.srt.observability.scheduler_stage_metrics import (
|
||||
SCHEDULER_STAGE_GET_NEXT_BATCH,
|
||||
@@ -565,40 +563,6 @@ class PrefillBootstrapQueue:
|
||||
self.kv_manager.register_buffer_to_engine()
|
||||
|
||||
|
||||
def should_async_state_payload(
|
||||
*,
|
||||
last_chunk: bool,
|
||||
supports_async_state_indices: bool,
|
||||
kv_shard_size: int,
|
||||
enable_all_cp_ranks_for_transfer: bool,
|
||||
enable_dsa_cache_layer_split: bool,
|
||||
) -> bool:
|
||||
"""Gate for the first async state-payload version (SWA-only).
|
||||
|
||||
``kv_shard_size > 1`` and all-CP-rank transfer **without DSA cache layer
|
||||
split** keep the old synchronous path: their indices are filtered before
|
||||
the transfer event, and deferring that to the worker would widen the
|
||||
pre-event shard-filter risk tracked by #45492.
|
||||
"""
|
||||
if not last_chunk or not supports_async_state_indices:
|
||||
return False
|
||||
if kv_shard_size != 1:
|
||||
return False
|
||||
if enable_all_cp_ranks_for_transfer and not enable_dsa_cache_layer_split:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def require_async_state_event(
|
||||
async_state: bool, segment_is_last: bool, indices_event
|
||||
) -> None:
|
||||
"""Fail loudly if an async state payload has no event to protect its copy."""
|
||||
if async_state and segment_is_last and indices_event is None:
|
||||
raise RuntimeError(
|
||||
"async state payload requires an indices_event for the final segment"
|
||||
)
|
||||
|
||||
|
||||
class SchedulerDisaggregationPrefillMixin:
|
||||
"""
|
||||
Mixin for Scheduler to handle disaggregation prefill
|
||||
@@ -652,15 +616,11 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
self.attn_tp_cpu_group,
|
||||
)
|
||||
failed = set()
|
||||
bootstrap_candidates = len(candidates)
|
||||
pending_bootstrap = 0
|
||||
for req, poll in zip(candidates, polls):
|
||||
if poll == KVPoll.Failed:
|
||||
self.handle_bootstrap_failure(req)
|
||||
failed.add(req)
|
||||
bootstrap_candidates -= 1
|
||||
continue
|
||||
if (
|
||||
elif (
|
||||
poll == KVPoll.WaitingForInput
|
||||
and req.pending_bootstrap
|
||||
and not should_force_retry(req)
|
||||
@@ -669,10 +629,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
# finalize cannot fail here; if it ever does, the request stays
|
||||
# pending and the post-forward check resolves it.
|
||||
self.disagg_prefill_bootstrap_queue.finalize_bootstrap(req)
|
||||
pending_bootstrap += int(req.pending_bootstrap)
|
||||
batch_telemetry.set_bootstrap_pending(
|
||||
pending_bootstrap, bootstrap_candidates
|
||||
)
|
||||
if failed:
|
||||
self.waiting_queue = [
|
||||
req for req in self.waiting_queue if req not in failed
|
||||
@@ -691,71 +647,32 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
last_batch: Optional[ScheduleBatch],
|
||||
should_retry_storage_prefetch: bool = True,
|
||||
) -> NextBatchPlan:
|
||||
bt_pass = batch_telemetry.begin_pass()
|
||||
if bt_pass is not None:
|
||||
# Capture before process_prefill_chunk()/admission can consume it:
|
||||
# the last chunk of a resumed request clears self.chunked_req.
|
||||
batch_telemetry.note_had_chunked(self.chunked_req is not None)
|
||||
|
||||
if bt_pass is not None:
|
||||
_bt_t = time.monotonic()
|
||||
self.process_pending_chunked_abort()
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_pending_abort(time.monotonic() - _bt_t)
|
||||
_bt_t = time.monotonic()
|
||||
self._process_hicache_events(
|
||||
should_retry_storage_prefetch=should_retry_storage_prefetch
|
||||
)
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_hicache(time.monotonic() - _bt_t)
|
||||
|
||||
# HACK (byronhsu): reset the batch_is_full flag because we never enter update_running_batch which resets it
|
||||
# Otherwise, it hangs under high concurrency
|
||||
running_batch.batch_is_full = False
|
||||
|
||||
if bt_pass is not None:
|
||||
_bt_t = time.monotonic()
|
||||
self.resolve_waiting_queue_bootstrap()
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_bootstrap(time.monotonic() - _bt_t)
|
||||
|
||||
# Skip repeated chunk processing while polling a pending batch.
|
||||
_chunk_t0 = time.monotonic() if bt_pass is not None else None
|
||||
_had_chunk = self.chunked_req is not None
|
||||
if last_batch is None or not last_batch.prefill_chunk_processed:
|
||||
self.process_prefill_chunk(
|
||||
last_batch=last_batch, running_batch=running_batch
|
||||
)
|
||||
if last_batch is not None:
|
||||
last_batch.prefill_chunk_processed = True
|
||||
if _chunk_t0 is not None:
|
||||
batch_telemetry.add_chunk(
|
||||
time.monotonic() - _chunk_t0, 1 if _had_chunk else 0
|
||||
)
|
||||
|
||||
_nb_t0 = time.monotonic() if bt_pass is not None else None
|
||||
prefill_plan = self.get_new_batch_prefill(running_batch)
|
||||
batch = prefill_plan.batch_to_run
|
||||
running_batch = prefill_plan.running_batch
|
||||
if _nb_t0 is not None:
|
||||
batch_telemetry.add_new_batch(time.monotonic() - _nb_t0)
|
||||
if bt_pass is not None:
|
||||
# Drop any gather value left by an earlier pass; only a gather that
|
||||
# actually runs inside the call below may be attributed to this one.
|
||||
batch_telemetry.note_dp_gather(None)
|
||||
_bt_t = time.monotonic()
|
||||
batch = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(batch)
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
|
||||
|
||||
if batch:
|
||||
_ss_t0 = time.monotonic() if bt_pass is not None else None
|
||||
set_schedule_time_batch(batch)
|
||||
if _ss_t0 is not None:
|
||||
batch_telemetry.add_set_sched(time.monotonic() - _ss_t0)
|
||||
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.end_pass(dp_rank=get_parallel().attn_dp_rank)
|
||||
|
||||
return NextBatchPlan(batch_to_run=batch, running_batch=running_batch)
|
||||
|
||||
@@ -848,94 +765,10 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
batch = None
|
||||
batch_result = None
|
||||
|
||||
def _forward_active() -> bool:
|
||||
reporter = getattr(self, "metrics_reporter", None)
|
||||
timer = (
|
||||
getattr(reporter, "forward_pass_device_timer", None)
|
||||
if reporter is not None
|
||||
else None
|
||||
)
|
||||
return timer.is_active() if timer is not None else False
|
||||
|
||||
def _pre_has_work() -> bool:
|
||||
bootstrap_queue = getattr(
|
||||
self.disagg_prefill_bootstrap_queue, "queue", None
|
||||
)
|
||||
return bool(
|
||||
self.last_batch is not None
|
||||
or len(self.result_queue) > 0
|
||||
or len(self.waiting_queue) > 0
|
||||
or len(self._deferred_input_requests) > 0
|
||||
or self._engine_paused
|
||||
or (
|
||||
self.disagg_prefill_inflight_queue is not None
|
||||
and len(self.disagg_prefill_inflight_queue) > 0
|
||||
)
|
||||
or (bootstrap_queue is not None and len(bootstrap_queue) > 0)
|
||||
)
|
||||
|
||||
def _loop_state(
|
||||
waiting_for_result: bool = False, paused: bool = False
|
||||
) -> dict:
|
||||
bootstrap_queue = getattr(
|
||||
self.disagg_prefill_bootstrap_queue, "queue", None
|
||||
)
|
||||
inflight = self.disagg_prefill_inflight_queue
|
||||
return {
|
||||
"waiting": len(self.waiting_queue),
|
||||
"result_queue": len(self.result_queue),
|
||||
"result_limit": 2,
|
||||
"inflight": len(inflight) if inflight is not None else 0,
|
||||
"bootstrap": len(bootstrap_queue) if bootstrap_queue is not None else 0,
|
||||
"deferred": len(self._deferred_input_requests),
|
||||
"paused": bool(paused or self._engine_paused),
|
||||
"waiting_for_result": bool(waiting_for_result),
|
||||
"skip_batch_creation": bool(skip_batch_creation),
|
||||
"forward_active": _forward_active(),
|
||||
}
|
||||
|
||||
def _end_loop(
|
||||
lt,
|
||||
has_batch: bool,
|
||||
loop_batch,
|
||||
waiting_for_result: bool = False,
|
||||
paused: bool = False,
|
||||
) -> None:
|
||||
if lt is None:
|
||||
return
|
||||
try:
|
||||
batch_telemetry.end_loop(
|
||||
lt,
|
||||
dp_rank=get_parallel().attn_dp_rank,
|
||||
has_batch=has_batch,
|
||||
batch=loop_batch,
|
||||
state=_loop_state(
|
||||
waiting_for_result=waiting_for_result, paused=paused
|
||||
),
|
||||
)
|
||||
except Exception:
|
||||
# Telemetry must never break the scheduler loop.
|
||||
pass
|
||||
|
||||
while True:
|
||||
if self.gracefully_exit:
|
||||
break
|
||||
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline
|
||||
|
||||
fwd_timeline.drain()
|
||||
except Exception:
|
||||
pass
|
||||
_lt = None
|
||||
if batch_telemetry.loop_accounting_enabled():
|
||||
_lt = batch_telemetry.begin_loop(
|
||||
"prefill",
|
||||
pre_has_work=_pre_has_work(),
|
||||
forward_active=_forward_active,
|
||||
)
|
||||
loop_batch = None
|
||||
|
||||
# Stop dispatching at pause requests while waiting for a batch result.
|
||||
received_inputs = self.ingest_requests(
|
||||
stop_at_pause=waiting_for_batch_result
|
||||
@@ -943,7 +776,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
if self._engine_paused:
|
||||
self._record_scheduler_state_for_paused_engine()
|
||||
self._yield_gil_if_needed()
|
||||
_end_loop(_lt, False, None, paused=True)
|
||||
continue
|
||||
bootstrapped_reqs = self.disagg_prefill_bootstrap_queue.pop_bootstrapped()
|
||||
self.waiting_queue.extend(bootstrapped_reqs)
|
||||
@@ -951,14 +783,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
if not waiting_for_batch_result or received_inputs or bootstrapped_reqs:
|
||||
skip_batch_creation = False
|
||||
|
||||
# Preserve the launch decision before processing a result mutates the
|
||||
# queues; end-of-loop depths cannot explain why this loop did not launch.
|
||||
if _lt is not None:
|
||||
if len(self.result_queue) >= 2:
|
||||
batch_telemetry.note_launch_blocked(_lt, "result_backpressure")
|
||||
elif self._deferred_input_requests:
|
||||
batch_telemetry.note_launch_blocked(_lt, "deferred_input")
|
||||
|
||||
# Keep at most two batches in flight.
|
||||
if (
|
||||
len(self.result_queue) < 2
|
||||
@@ -978,7 +802,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
batch = self.ngram_embedding_manager.prepare_for_forward(
|
||||
batch, chunked_req=self.chunked_req
|
||||
)
|
||||
loop_batch = batch
|
||||
self.cur_batch_for_debug = batch
|
||||
# Skip batch creation next round to avoid repeated preparation failures.
|
||||
skip_batch_creation = batch is None
|
||||
@@ -987,31 +810,8 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
if batch:
|
||||
if self.enable_staging:
|
||||
self.maybe_prefetch_staging_for_batch(batch)
|
||||
_rb_t0 = (
|
||||
time.time_ns()
|
||||
if _lt is not None and _lt.span is not None
|
||||
else 0
|
||||
)
|
||||
batch_result = self.run_batch(batch)
|
||||
if _rb_t0:
|
||||
try:
|
||||
batch_telemetry.mark_run_batch(
|
||||
_rb_t0,
|
||||
time.time_ns(),
|
||||
{
|
||||
"sglang.forward_mode": str(batch.forward_mode),
|
||||
"sglang.bs": batch.batch_size(),
|
||||
"sglang.num_tokens": int(
|
||||
getattr(batch, "extend_num_tokens", 0) or 0
|
||||
),
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
# WAR barrier is outside run_batch's timing boundary, so it is
|
||||
# recorded as its own loop phase (design §4.3).
|
||||
with batch_telemetry.detail_phase("war_barrier"):
|
||||
self._apply_war_barrier()
|
||||
self._apply_war_barrier()
|
||||
self.result_queue.append((batch.copy(), batch_result))
|
||||
else:
|
||||
batch_result = None
|
||||
@@ -1028,67 +828,24 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
):
|
||||
waiting_for_batch_result = True
|
||||
self._yield_gil_if_needed()
|
||||
_end_loop(
|
||||
_lt,
|
||||
loop_batch is not None,
|
||||
loop_batch,
|
||||
waiting_for_result=True,
|
||||
)
|
||||
continue
|
||||
# Process the oldest batch result.
|
||||
oldest_batch, oldest_batch_result = self.result_queue.popleft()
|
||||
_pr_t0 = (
|
||||
time.time_ns() if _lt is not None and _lt.span is not None else 0
|
||||
)
|
||||
# Scope KV phases to this call: sends issued later in the loop
|
||||
# (inflight-queue pending bootstrap) must not be charged here.
|
||||
_proc_kv_phases = (
|
||||
"kv_state_payload",
|
||||
"kv_page_indices",
|
||||
"kv_sender_send",
|
||||
)
|
||||
_proc_detail = batch_telemetry.loop_detail_enabled()
|
||||
_proc_kv_before = (
|
||||
batch_telemetry.detail_phase_total_ns(_proc_kv_phases)
|
||||
if _proc_detail
|
||||
else 0
|
||||
)
|
||||
self.process_batch_result(oldest_batch, oldest_batch_result)
|
||||
_pr_attrs = {"sglang.n_reqs": len(oldest_batch.reqs)}
|
||||
if _proc_detail:
|
||||
_pr_attrs["sglang.process_kv_ms"] = (
|
||||
batch_telemetry.detail_phase_total_ns(_proc_kv_phases)
|
||||
- _proc_kv_before
|
||||
) / 1e6
|
||||
if _pr_t0:
|
||||
try:
|
||||
batch_telemetry.mark_process(
|
||||
_pr_t0,
|
||||
time.time_ns(),
|
||||
_pr_attrs,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
elif batch is None:
|
||||
# When the server is idle, do self-check and re-init some states
|
||||
self.on_idle()
|
||||
|
||||
# `loop other` (outside run_batch/process_result/get_next) hosts the
|
||||
# inflight poll + release_kv_cache(checkpoint/insert) and the sample
|
||||
# launch. Timed as its own phases so the loop-other residual shrinks.
|
||||
with batch_telemetry.detail_phase("inflight_poll"):
|
||||
self.process_disagg_prefill_inflight_queue()
|
||||
self.process_disagg_prefill_inflight_queue()
|
||||
|
||||
# Run sample of the current batch
|
||||
# It depends on the result of the last batch (e.g., grammar), so we run it after the last batch is processed.
|
||||
with batch_telemetry.detail_phase("launch_sample"):
|
||||
self.launch_batch_sample_if_needed(batch_result, batch)
|
||||
self.launch_batch_sample_if_needed(batch_result, batch)
|
||||
|
||||
# Update last_batch and scheduler status
|
||||
self._sched_idled = batch is None
|
||||
self.last_batch = batch
|
||||
waiting_for_batch_result = False
|
||||
_end_loop(_lt, loop_batch is not None, loop_batch)
|
||||
|
||||
def is_disagg_prefill_batch_result_ready(
|
||||
self: Scheduler,
|
||||
@@ -1140,11 +897,7 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
)
|
||||
|
||||
if copy_done is not None:
|
||||
with batch_telemetry.detail_phase("copy_done_wait"):
|
||||
copy_done.synchronize()
|
||||
_aux_t0 = (
|
||||
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
|
||||
)
|
||||
copy_done.synchronize()
|
||||
auxiliary_output_starts = (
|
||||
self.batch_result_processor.snapshot_auxiliary_output_starts(batch, result)
|
||||
)
|
||||
@@ -1190,29 +943,10 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
if extend_logprob_start_len < extend_input_len:
|
||||
logprob_pt += extend_input_len - extend_logprob_start_len
|
||||
|
||||
# Auxiliary finalize / logits-to-CPU work is a contiguous host phase.
|
||||
if _aux_t0 is not None:
|
||||
batch_telemetry.detail_add(
|
||||
"auxiliary_and_logprob", _aux_t0, time.monotonic_ns()
|
||||
)
|
||||
# Per-request result handling is everything in this loop except the KV
|
||||
# phases; the delta below keeps `continue` paths accounted for without
|
||||
# measuring each branch.
|
||||
_pr_loop_t0 = (
|
||||
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
|
||||
)
|
||||
_kv_phases = ("kv_state_payload", "kv_page_indices", "kv_sender_send")
|
||||
_kv_before = (
|
||||
batch_telemetry.detail_phase_total_ns(_kv_phases)
|
||||
if _pr_loop_t0 is not None
|
||||
else 0
|
||||
)
|
||||
|
||||
for i, (req, next_token_id) in enumerate(
|
||||
zip(batch.reqs, next_token_ids, strict=True)
|
||||
):
|
||||
if req.inflight_middle_chunks <= 0:
|
||||
batch_telemetry.detail_count("n_last_reqs")
|
||||
req.time_stats.set_prefill_finished_time()
|
||||
|
||||
# Its KV never landed: abort instead of sending it to decode.
|
||||
@@ -1273,7 +1007,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
|
||||
self.checkpoint_disagg_prefill(req)
|
||||
self.disagg_prefill_inflight_queue.append(req)
|
||||
batch_telemetry.note_request_completed("prefill")
|
||||
if self.spec_algorithm.is_eagle() and draft_input is not None:
|
||||
req.output_topk_p = draft_input.topk_p[i]
|
||||
req.output_topk_index = draft_input.topk_index[i]
|
||||
@@ -1317,7 +1050,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
|
||||
else:
|
||||
# being chunked reqs' prefill is not finished
|
||||
batch_telemetry.detail_count("n_middle_reqs")
|
||||
req.inflight_middle_chunks -= 1
|
||||
|
||||
# Still chunking iff its next chunk was launched: either it is
|
||||
@@ -1368,27 +1100,12 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
self.send_kv_chunk(req, last_chunk=False, end_idx=req.tmp_end_idx)
|
||||
req.time_stats.set_last_chunked_prefill_finish_time()
|
||||
|
||||
if _pr_loop_t0 is not None:
|
||||
_kv_ns = batch_telemetry.detail_phase_total_ns(_kv_phases) - _kv_before
|
||||
batch_telemetry.detail_add(
|
||||
"per_request_result",
|
||||
_pr_loop_t0 + max(0, _kv_ns),
|
||||
time.monotonic_ns(),
|
||||
)
|
||||
|
||||
if auxiliary_output is not None:
|
||||
_aux2_t0 = (
|
||||
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
|
||||
)
|
||||
self.batch_result_processor.consume_auxiliary_output(
|
||||
batch,
|
||||
auxiliary_output,
|
||||
auxiliary_output_starts,
|
||||
)
|
||||
if _aux2_t0 is not None:
|
||||
batch_telemetry.detail_add(
|
||||
"auxiliary_and_logprob", _aux2_t0, time.monotonic_ns()
|
||||
)
|
||||
|
||||
if aborted_reqs:
|
||||
self.output_streamer.stream_output(
|
||||
@@ -1397,13 +1114,12 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
)
|
||||
|
||||
can_run_cuda_graph = result.can_run_cuda_graph
|
||||
with batch_telemetry.detail_phase("report_stats"):
|
||||
self.metrics_reporter.report_prefill_stats(
|
||||
batch=batch,
|
||||
prefill_stats=batch.prefill_stats,
|
||||
can_run_cuda_graph=can_run_cuda_graph,
|
||||
dp_cooperation_info=batch.dp_cooperation_info,
|
||||
)
|
||||
self.metrics_reporter.report_prefill_stats(
|
||||
batch=batch,
|
||||
prefill_stats=batch.prefill_stats,
|
||||
can_run_cuda_graph=can_run_cuda_graph,
|
||||
dp_cooperation_info=batch.dp_cooperation_info,
|
||||
)
|
||||
self.maybe_send_health_check_signal()
|
||||
|
||||
@scheduler_stage_method(SCHEDULER_STAGE_PROCESS_QUEUE)
|
||||
@@ -1484,44 +1200,15 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
if not isinstance(req.finished_reason, FINISH_ABORT):
|
||||
req.finished_reason = FINISH_LENGTH(length=0)
|
||||
# unlock the tree
|
||||
_release_t0 = (
|
||||
time.monotonic_ns()
|
||||
if batch_telemetry.loop_detail_enabled()
|
||||
else None
|
||||
)
|
||||
release_kv_cache(req, self.tree_cache, checkpoint=True)
|
||||
self.tree_cache.finish(
|
||||
req.cache_request_handle, CacheRequestOutcome.SUCCESS
|
||||
)
|
||||
if _release_t0 is not None:
|
||||
# Nested in inflight_poll: checkpoint + radix/hicache insert.
|
||||
batch_telemetry.detail_add(
|
||||
"inflight_release", _release_t0, time.monotonic_ns()
|
||||
)
|
||||
# Consume the join summary before clear() releases the room.
|
||||
_join_stats = None
|
||||
if batch_telemetry.mooncake_detail_enabled():
|
||||
_pop_stats = getattr(
|
||||
req.disagg_kv_sender, "pop_chunk_join_stats", None
|
||||
)
|
||||
_join_stats = _pop_stats() if _pop_stats is not None else None
|
||||
# FIXME: clean up req's data in transfer engine
|
||||
req.disagg_kv_sender.clear()
|
||||
done_reqs.append(req)
|
||||
req.time_stats.set_prefill_kv_transfer_finish_time()
|
||||
if batch_telemetry.mooncake_detail_enabled():
|
||||
batch_telemetry.emit_request_kv_join(
|
||||
req=req,
|
||||
stats=_join_stats,
|
||||
dp_rank=get_parallel().attn_dp_rank,
|
||||
)
|
||||
elif poll == KVPoll.Failed:
|
||||
if batch_telemetry.mooncake_detail_enabled():
|
||||
_pop_stats = getattr(
|
||||
req.disagg_kv_sender, "pop_chunk_join_stats", None
|
||||
)
|
||||
if _pop_stats is not None:
|
||||
_pop_stats()
|
||||
self.handle_inflight_transfer_failure(req)
|
||||
done_reqs.append(req)
|
||||
else:
|
||||
@@ -1780,21 +1467,14 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
ev = torch.cuda.Event()
|
||||
ev.record(self.forward_stream)
|
||||
req.disagg_kv_sender._early_send_wait_event = ev
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
batch_telemetry.detail_count("early_send_reqs")
|
||||
self.send_kv_chunk(
|
||||
req, last_chunk=False, end_idx=cached_end, send_kind="cached_prefix"
|
||||
)
|
||||
self.send_kv_chunk(req, last_chunk=False, end_idx=cached_end)
|
||||
|
||||
def send_kv_chunk(
|
||||
self,
|
||||
req: Req,
|
||||
last_chunk: bool = False,
|
||||
end_idx: Optional[int] = None,
|
||||
send_kind: Optional[str] = None,
|
||||
) -> None:
|
||||
if send_kind is None:
|
||||
send_kind = "last" if last_chunk else "middle"
|
||||
computer: Optional[KvChecksumComputer] = self.kv_checksum_computer
|
||||
if last_chunk and computer is not None:
|
||||
if is_health_check_req(req):
|
||||
@@ -1806,21 +1486,13 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
state_indices = state_indices_for_request(self, req, end_idx)
|
||||
value = computer.compute(page_indices_gpu, state_indices)
|
||||
self.disagg_metadata_buffers.set_kv_checksum(req, value)
|
||||
_kv_t0 = time.time_ns() if batch_telemetry.kv_send_span_enabled() else 0
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
batch_telemetry.detail_count("n_kv_calls")
|
||||
self._send_kv_chunk(
|
||||
req, last_chunk=last_chunk, end_idx=end_idx, send_kind=send_kind
|
||||
)
|
||||
if _kv_t0:
|
||||
batch_telemetry.add_kv_send(_kv_t0, time.time_ns())
|
||||
self._send_kv_chunk(req, last_chunk=last_chunk, end_idx=end_idx)
|
||||
|
||||
def _send_kv_chunk(
|
||||
self: Scheduler,
|
||||
req: Req,
|
||||
last_chunk: bool = False,
|
||||
end_idx: Optional[int] = None,
|
||||
send_kind: str = "last",
|
||||
) -> None:
|
||||
"""
|
||||
Send a prefilled chunk to the decode server
|
||||
@@ -1854,27 +1526,7 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
)
|
||||
return
|
||||
|
||||
# First async state-payload version: SWA only, with the shard/CP guards
|
||||
# in should_async_state_payload(). Other builders stay synchronous.
|
||||
sender = req.disagg_kv_sender
|
||||
async_state = should_async_state_payload(
|
||||
last_chunk=last_chunk,
|
||||
supports_async_state_indices=getattr(
|
||||
sender, "supports_async_state_indices", False
|
||||
),
|
||||
kv_shard_size=getattr(sender.kv_mgr, "kv_shard_size", 1),
|
||||
enable_all_cp_ranks_for_transfer=getattr(
|
||||
sender.kv_mgr, "enable_all_cp_ranks_for_transfer", False
|
||||
),
|
||||
enable_dsa_cache_layer_split=get_parallel().enable_dsa_cache_layer_split,
|
||||
)
|
||||
|
||||
state_indices: Optional[List] = None
|
||||
_state_tok = (
|
||||
batch_telemetry.detail_begin("kv_state_payload." + send_kind)
|
||||
if last_chunk
|
||||
else None
|
||||
)
|
||||
if last_chunk:
|
||||
self.disagg_metadata_buffers.set_buf(req)
|
||||
|
||||
@@ -1920,10 +1572,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
window_kv_indices_full
|
||||
)
|
||||
)
|
||||
if async_state:
|
||||
return kv_to_page_indices_pinned(
|
||||
window_kv_indices_swa, page_size
|
||||
)
|
||||
return kv_to_page_indices(window_kv_indices_swa, page_size)
|
||||
|
||||
def _full_kv_pages_payload():
|
||||
@@ -2017,12 +1665,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
state_indices = [
|
||||
payloads[st]() if st in payloads else None for st in state_types
|
||||
]
|
||||
if _state_tok is not None:
|
||||
batch_telemetry.detail_attr(
|
||||
"state_types",
|
||||
",".join(sorted(str(getattr(st, "name", st)) for st in state_types)),
|
||||
)
|
||||
batch_telemetry.detail_end("kv_state_payload." + send_kind, _state_tok)
|
||||
|
||||
transfer_chunk_tokens = req.disagg_kv_sender.get_max_transfer_tokens()
|
||||
if self.enable_staging:
|
||||
@@ -2043,16 +1685,7 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
else:
|
||||
segments = [(start_idx, end_idx)]
|
||||
|
||||
# Classify the chunk for detail accounting (cached_prefix/middle/last).
|
||||
# Set on the sender so backend send() signatures stay unchanged.
|
||||
req.disagg_kv_sender._send_kind = send_kind
|
||||
|
||||
for seg_start, seg_end in segments:
|
||||
_seg_t0 = (
|
||||
time.monotonic_ns()
|
||||
if batch_telemetry.loop_detail_enabled()
|
||||
else None
|
||||
)
|
||||
is_final_segment = seg_end == end_idx
|
||||
raw_kv_indices = self.req_to_token_pool.req_to_token[
|
||||
req.kv.req_pool_idx, seg_start:seg_end
|
||||
@@ -2064,6 +1697,7 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
raw_kv_indices
|
||||
)
|
||||
)
|
||||
sender = req.disagg_kv_sender
|
||||
indices_event = None
|
||||
if getattr(sender, "supports_host_tensor_indices", False) and not (
|
||||
sender.kv_mgr.enable_all_cp_ranks_for_transfer
|
||||
@@ -2083,23 +1717,10 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
indices_event.record(torch.cuda.current_stream())
|
||||
else:
|
||||
page_indices = kv_to_page_indices(kv_indices, page_size)
|
||||
if _seg_t0 is not None:
|
||||
batch_telemetry.detail_add(
|
||||
"kv_page_indices." + send_kind, _seg_t0, time.monotonic_ns()
|
||||
)
|
||||
batch_telemetry.detail_count("n_pages", len(page_indices))
|
||||
segment_is_last = last_chunk and is_final_segment
|
||||
if not sender.should_send_kv_chunk(len(page_indices), segment_is_last):
|
||||
continue
|
||||
require_async_state_event(
|
||||
async_state, segment_is_last, indices_event
|
||||
)
|
||||
send_state_indices = state_indices if segment_is_last else None
|
||||
_send_t0 = (
|
||||
time.monotonic_ns()
|
||||
if batch_telemetry.loop_detail_enabled()
|
||||
else None
|
||||
)
|
||||
if indices_event is not None:
|
||||
sender.send(
|
||||
page_indices,
|
||||
@@ -2113,11 +1734,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
send_state_indices,
|
||||
num_kv_tokens=seg_end - seg_start,
|
||||
)
|
||||
if _send_t0 is not None:
|
||||
batch_telemetry.detail_add(
|
||||
"kv_sender_send." + send_kind, _send_t0, time.monotonic_ns()
|
||||
)
|
||||
batch_telemetry.detail_count("n_segments")
|
||||
req.start_send_idx = end_idx
|
||||
# A last chunk needs no entry: every `last_chunk=True` call site has
|
||||
# already put the request on `disagg_prefill_inflight_queue`.
|
||||
@@ -2128,12 +1744,6 @@ class SchedulerDisaggregationPrefillMixin:
|
||||
|
||||
def optimistic_release_and_requeue(self: Scheduler, req: Req) -> None:
|
||||
"""Release KV cache and requeue an optimistic prefill request."""
|
||||
if batch_telemetry.mooncake_detail_enabled():
|
||||
# A retry may reuse the same room; stale chunk stats would
|
||||
# otherwise be merged into the next attempt's join summary.
|
||||
_reset = getattr(req.disagg_kv_sender, "reset_chunk_detail_state", None)
|
||||
if _reset is not None:
|
||||
_reset()
|
||||
max_attempts = get_disagg().optimistic_prefill_attempts
|
||||
uses_write_through_cache = _uses_write_through_cache(self.tree_cache)
|
||||
if not uses_write_through_cache:
|
||||
|
||||
@@ -56,7 +56,6 @@ class AnthropicUsage(BaseModel):
|
||||
class TextBlock(BaseModel):
|
||||
type: Literal["text"] = "text"
|
||||
text: str
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class ImageBlock(BaseModel):
|
||||
@@ -92,50 +91,6 @@ class ToolReferenceBlock(BaseModel):
|
||||
id: Optional[str] = None
|
||||
|
||||
|
||||
class McpToolReference(BaseModel):
|
||||
type: Literal["mcp_tool_reference"]
|
||||
server_name: str
|
||||
name: str
|
||||
|
||||
|
||||
class McpToolsetReference(BaseModel):
|
||||
type: Literal["mcp_toolset_reference"]
|
||||
server_name: str
|
||||
|
||||
|
||||
class ToolDefinition(BaseModel):
|
||||
"""A tool given by value: `definition` is a `tools` entry."""
|
||||
|
||||
type: Literal["tool_definition"]
|
||||
definition: dict[str, Any]
|
||||
|
||||
|
||||
class ToolAdditionBlock(BaseModel):
|
||||
"""Anthropic beta block: a ``system`` message offers a tool from its position
|
||||
on. Rendered through deepseek-recipe, which reads only a ``tool_reference``
|
||||
target."""
|
||||
|
||||
type: Literal["tool_addition"] = "tool_addition"
|
||||
tool: Annotated[
|
||||
Union[
|
||||
ToolReferenceBlock, McpToolReference, McpToolsetReference, ToolDefinition
|
||||
],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class ToolRemovalBlock(BaseModel):
|
||||
"""Anthropic beta block: a ``system`` message withdraws a tool."""
|
||||
|
||||
type: Literal["tool_removal"] = "tool_removal"
|
||||
tool: Annotated[
|
||||
Union[ToolReferenceBlock, McpToolReference, McpToolsetReference],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class SearchResultBlock(BaseModel):
|
||||
type: Literal["search_result"] = "search_result"
|
||||
# ``source`` here is a URL/identifier string (unlike ImageBlock.source).
|
||||
@@ -162,8 +117,6 @@ AnthropicContentBlock = Annotated[
|
||||
ToolUseBlock,
|
||||
ToolResultBlock,
|
||||
ToolReferenceBlock,
|
||||
ToolAdditionBlock,
|
||||
ToolRemovalBlock,
|
||||
SearchResultBlock,
|
||||
ThinkingBlock,
|
||||
RedactedThinkingBlock,
|
||||
@@ -175,8 +128,6 @@ AnthropicContentBlock = Annotated[
|
||||
class AnthropicMessage(BaseModel):
|
||||
role: Literal["user", "assistant", "system"]
|
||||
content: Union[str, list[AnthropicContentBlock]]
|
||||
clear_at: Optional[Literal["next_user_message", "never"]] = None
|
||||
output_config: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
# ---------- Tools (discriminated by ``type`` family) ----------
|
||||
|
||||
@@ -20,10 +20,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
AnthropicContentBlock,
|
||||
AnthropicCountTokensRequest,
|
||||
AnthropicCountTokensResponse,
|
||||
AnthropicCustomTool,
|
||||
AnthropicError,
|
||||
AnthropicErrorResponse,
|
||||
AnthropicMessage,
|
||||
AnthropicMessageEndDelta,
|
||||
AnthropicMessagesRequest,
|
||||
AnthropicMessagesResponse,
|
||||
@@ -34,8 +32,6 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
ContentBlockStopEvent,
|
||||
ErrorEvent,
|
||||
InputJsonDelta,
|
||||
McpToolReference,
|
||||
McpToolsetReference,
|
||||
MessageDeltaEvent,
|
||||
MessageStartEvent,
|
||||
MessageStopEvent,
|
||||
@@ -44,8 +40,6 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
TextDelta,
|
||||
ThinkingBlock,
|
||||
ThinkingDelta,
|
||||
ToolDefinition,
|
||||
ToolReferenceBlock,
|
||||
ToolUseBlock,
|
||||
is_server_tool,
|
||||
)
|
||||
@@ -158,207 +152,6 @@ def _extract_system_text(
|
||||
return "\n".join(texts) if texts else None
|
||||
|
||||
|
||||
def _block_types(request: AnthropicMessagesRequest) -> set[str]:
|
||||
"""The block types of the request's system and messages, tool result contents
|
||||
included."""
|
||||
types = set()
|
||||
for content in [request.system, *(message.content for message in request.messages)]:
|
||||
for block in content if isinstance(content, list) else []:
|
||||
types.add(block.type)
|
||||
if block.type == "tool_result" and isinstance(block.content, list):
|
||||
types.update(item.type for item in block.content)
|
||||
return types
|
||||
|
||||
|
||||
def _without_cleared_messages(
|
||||
request: AnthropicMessagesRequest,
|
||||
) -> AnthropicMessagesRequest:
|
||||
"""`request` without the turn-scoped system messages (`clear_at:
|
||||
"next_user_message"`) that a later user message has cleared."""
|
||||
last_user = max(
|
||||
(
|
||||
index
|
||||
for index, message in enumerate(request.messages)
|
||||
if message.role == "user"
|
||||
),
|
||||
default=-1,
|
||||
)
|
||||
rendered = []
|
||||
for index, message in enumerate(request.messages):
|
||||
if message.clear_at is not None and message.role != "system":
|
||||
raise ValueError(
|
||||
f"messages.{index}.clear_at: clear_at is only permitted on role "
|
||||
"'system' messages"
|
||||
)
|
||||
if message.clear_at == "next_user_message":
|
||||
_check_turn_scoped(index=index, message=message)
|
||||
_check_turn_scoped_placement(index=index, messages=request.messages)
|
||||
if index < last_user:
|
||||
continue
|
||||
rendered.append(message)
|
||||
return request.model_copy(update={"messages": rendered})
|
||||
|
||||
|
||||
def _check_turn_scoped(index: int, message: AnthropicMessage) -> None:
|
||||
scope = "a turn-scoped system message (clear_at: 'next_user_message')"
|
||||
if message.output_config is not None:
|
||||
raise ValueError(f"messages.{index}: output_config is not permitted on {scope}")
|
||||
for position, block in enumerate(
|
||||
message.content if isinstance(message.content, list) else []
|
||||
):
|
||||
if block.type != "text":
|
||||
raise ValueError(
|
||||
f"messages.{index}: a turn-scoped system message supports text "
|
||||
"blocks only (clear_at: 'next_user_message')"
|
||||
)
|
||||
if block.cache_control is not None:
|
||||
raise ValueError(
|
||||
f"messages.{index}.content.{position}: cache_control is not "
|
||||
f"permitted on {scope}"
|
||||
)
|
||||
|
||||
|
||||
def _check_turn_scoped_placement(index: int, messages: list[AnthropicMessage]) -> None:
|
||||
# Consecutive system messages are placed as one group.
|
||||
before = next(
|
||||
(m.role for m in reversed(messages[:index]) if m.role != "system"), None
|
||||
)
|
||||
after = next((m.role for m in messages[index + 1 :] if m.role != "system"), None)
|
||||
if before != "user" or after == "user":
|
||||
raise ValueError(
|
||||
f"messages.{index}: a turn-scoped system message (clear_at: "
|
||||
"'next_user_message') must follow a user message and either end "
|
||||
"messages or precede an assistant message"
|
||||
)
|
||||
|
||||
|
||||
def _declare_inline_tools(
|
||||
request: AnthropicMessagesRequest,
|
||||
) -> AnthropicMessagesRequest:
|
||||
"""`request` with each custom tool a `tool_addition` block defines by value
|
||||
declared in `tools` as deferred, and the block naming it by reference. A
|
||||
definition of a tool already offered unchanged is dropped."""
|
||||
tools = list(request.tools or [])
|
||||
declared = {tool.name: tool for tool in tools}
|
||||
offered = {tool.name for tool in tools if not tool.defer_loading}
|
||||
unresolved: dict[str, str] = {}
|
||||
messages = []
|
||||
for index, message in enumerate(request.messages):
|
||||
if isinstance(message.content, str):
|
||||
messages.append(message)
|
||||
continue
|
||||
blocks = []
|
||||
for position, block in enumerate(message.content):
|
||||
where = f"messages.{index}.content.{position}"
|
||||
for name in _referenced_tool_names(block):
|
||||
if name not in declared:
|
||||
unresolved.setdefault(name, where)
|
||||
if block.type == "tool_addition" and block.tool.type != "tool_reference":
|
||||
if message.role != "system":
|
||||
raise ValueError(
|
||||
f"{where}: tool_addition blocks can only be in system messages"
|
||||
)
|
||||
tool = _inline_custom_tool(where=where, target=block.tool)
|
||||
# Declaring the tool makes it resolvable from the first message on.
|
||||
if tool.name in unresolved:
|
||||
raise ValueError(
|
||||
f"{unresolved[tool.name]}: tool reference {tool.name!r} "
|
||||
f"precedes the tool_definition that defines it at {where}"
|
||||
)
|
||||
earlier = declared.setdefault(tool.name, tool)
|
||||
if earlier is tool:
|
||||
tools.append(tool)
|
||||
elif is_server_tool(earlier) or _definition(earlier) != _definition(
|
||||
tool
|
||||
):
|
||||
raise ValueError(
|
||||
f"{where}: tool_definition "
|
||||
f"reuses the name {tool.name!r} with a different "
|
||||
"definition; replacing a tool mid-conversation is not "
|
||||
"supported"
|
||||
)
|
||||
if tool.name in offered:
|
||||
continue
|
||||
offered.add(tool.name)
|
||||
block = block.model_copy(
|
||||
update={"tool": ToolReferenceBlock(name=tool.name)}
|
||||
)
|
||||
blocks.append(block)
|
||||
# recipe renders a system message left without blocks as an empty turn.
|
||||
if blocks or not message.content:
|
||||
messages.append(message.model_copy(update={"content": blocks}))
|
||||
return request.model_copy(update={"tools": tools, "messages": messages})
|
||||
|
||||
|
||||
def _referenced_tool_names(block: AnthropicContentBlock) -> list[Optional[str]]:
|
||||
if block.type == "tool_addition" and block.tool.type == "tool_reference":
|
||||
return [block.tool.name or block.tool.tool_name]
|
||||
if block.type == "tool_reference":
|
||||
return [block.tool_name or block.name]
|
||||
if block.type == "tool_result" and isinstance(block.content, list):
|
||||
return [name for item in block.content for name in _referenced_tool_names(item)]
|
||||
return []
|
||||
|
||||
|
||||
def _inline_custom_tool(
|
||||
where: str, target: Union[McpToolReference, McpToolsetReference, ToolDefinition]
|
||||
) -> AnthropicCustomTool:
|
||||
if target.type != "tool_definition":
|
||||
raise ValueError(
|
||||
f"{where}: tool_addition with a {target.type} is not supported; this "
|
||||
"server runs no MCP connector"
|
||||
)
|
||||
kind = target.definition.get("type")
|
||||
if kind not in (None, "custom"):
|
||||
raise ValueError(
|
||||
f"{where}: tool_definition supports custom tools only, got a "
|
||||
f"definition of type {kind!r}"
|
||||
)
|
||||
tool = AnthropicCustomTool.model_validate(target.definition)
|
||||
if tool.defer_loading:
|
||||
raise ValueError(
|
||||
f"{where}: a tool_definition with defer_loading is not supported; "
|
||||
"declare the tool in tools and add it by reference"
|
||||
)
|
||||
return tool.model_copy(update={"defer_loading": True})
|
||||
|
||||
|
||||
def _definition(tool: AnthropicCustomTool) -> dict[str, Any]:
|
||||
return tool.model_dump(exclude={"type", "defer_loading"})
|
||||
|
||||
|
||||
def _native_block(block: dict[str, Any]) -> dict[str, Any]:
|
||||
"""`block` with SGLang's accepted aliases spelled as deepseek-recipe reads them."""
|
||||
kind = block["type"]
|
||||
if kind == "tool_result":
|
||||
content = block.get("content")
|
||||
fields = {
|
||||
"type": kind,
|
||||
"tool_use_id": block.get("tool_use_id") or block.get("id"),
|
||||
"content": (
|
||||
[_native_block(item) for item in content]
|
||||
if isinstance(content, list)
|
||||
else content
|
||||
),
|
||||
"is_error": block.get("is_error"),
|
||||
}
|
||||
return {key: value for key, value in fields.items() if value is not None}
|
||||
if kind == "tool_reference":
|
||||
return {"type": kind, "tool_name": block.get("tool_name") or block.get("name")}
|
||||
if kind == "tool_addition":
|
||||
tool = block["tool"]
|
||||
name = tool.get("name") or tool.get("tool_name")
|
||||
return {"type": kind, "tool": {"type": "tool_reference", "name": name}}
|
||||
return block
|
||||
|
||||
|
||||
def _native_message(message: AnthropicMessage) -> dict[str, Any]:
|
||||
data = message.model_dump(exclude_none=True)
|
||||
if isinstance(data["content"], list):
|
||||
data["content"] = [_native_block(block) for block in data["content"]]
|
||||
return data
|
||||
|
||||
|
||||
def _wrap_sse_event(data: str, event_type: str) -> str:
|
||||
"""Format an Anthropic SSE event with event type and data lines."""
|
||||
return f"event: {event_type}\ndata: {data}\n\n"
|
||||
@@ -425,24 +218,6 @@ class AnthropicServing:
|
||||
self, anthropic_request: AnthropicMessagesRequest
|
||||
) -> ChatCompletionRequest:
|
||||
"""Convert an Anthropic Messages request to an OpenAI ChatCompletion request."""
|
||||
anthropic_request = _declare_inline_tools(
|
||||
_without_cleared_messages(anthropic_request)
|
||||
)
|
||||
block_types = _block_types(anthropic_request)
|
||||
if "tool_removal" in block_types:
|
||||
raise ValueError(
|
||||
"tool_removal blocks are not supported; a tool cannot be withdrawn "
|
||||
"mid-conversation"
|
||||
)
|
||||
native = self.openai_serving_chat.native_tool_loading and (
|
||||
bool(block_types & {"tool_reference", "tool_addition"})
|
||||
or any(
|
||||
not is_server_tool(tool) and tool.defer_loading
|
||||
for tool in anthropic_request.tools or []
|
||||
)
|
||||
)
|
||||
if not native and "tool_addition" in block_types:
|
||||
raise ValueError("tool_addition blocks are not supported by this model")
|
||||
openai_messages = []
|
||||
|
||||
def _convert_anthropic_image_source_to_openai_part(
|
||||
@@ -944,51 +719,8 @@ class AnthropicServing:
|
||||
elif chat_request.tools:
|
||||
chat_request.tool_choice = "auto"
|
||||
|
||||
if native:
|
||||
self._prepare_native_prompt(anthropic_request, chat_request)
|
||||
return chat_request
|
||||
|
||||
def _prepare_native_prompt(
|
||||
self,
|
||||
anthropic_request: AnthropicMessagesRequest,
|
||||
chat_request: ChatCompletionRequest,
|
||||
) -> None:
|
||||
"""Render the request's own wire form into `chat_request`, whose tools
|
||||
become the ones the history has loaded."""
|
||||
body = {
|
||||
"messages": [
|
||||
_native_message(message) for message in anthropic_request.messages
|
||||
],
|
||||
# recipe reads a tool with a `type` as a server tool.
|
||||
"tools": [
|
||||
tool.model_dump(exclude_none=True, exclude={"type"})
|
||||
for tool in anthropic_request.tools or []
|
||||
if not is_server_tool(tool)
|
||||
],
|
||||
}
|
||||
system = anthropic_request.system
|
||||
if system:
|
||||
body["system"] = (
|
||||
system
|
||||
if isinstance(system, str)
|
||||
else [block.model_dump(exclude_none=True) for block in system]
|
||||
)
|
||||
choice = anthropic_request.tool_choice
|
||||
if choice is not None and choice.type == "none":
|
||||
body["tool_choice"] = {"type": "none"}
|
||||
tools = self.openai_serving_chat.prepare_native_prompt(
|
||||
chat_request, protocol="messages", body=body
|
||||
).tools
|
||||
if (
|
||||
choice is not None
|
||||
and choice.type == "tool"
|
||||
and not any(tool.function.name == choice.name for tool in tools)
|
||||
):
|
||||
raise ValueError(
|
||||
f"tool_choice names {choice.name!r}, a deferred tool no "
|
||||
"tool_reference or tool_addition block has loaded"
|
||||
)
|
||||
|
||||
async def _handle_non_streaming(
|
||||
self,
|
||||
chat_request: ChatCompletionRequest,
|
||||
|
||||
@@ -3,7 +3,7 @@ concrete adapters load only when selected."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol, Union
|
||||
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol
|
||||
|
||||
import msgspec
|
||||
|
||||
@@ -15,27 +15,8 @@ if TYPE_CHECKING:
|
||||
ToolCall,
|
||||
)
|
||||
from sglang.srt.function_call.core_types import ToolCallItem
|
||||
from sglang.srt.multimodal.media_io import ImageData
|
||||
|
||||
EncodingBackend = Literal["builtin", "recipe"]
|
||||
HistoryToolCallArguments = Literal["strict", "lenient", "as_sent"]
|
||||
NativeProtocol = Literal["responses", "messages"]
|
||||
|
||||
|
||||
class NativePrompt(msgspec.Struct, frozen=True, kw_only=True):
|
||||
"""A Responses or Messages request rendered from its own wire form."""
|
||||
|
||||
prompt_ids: list[int]
|
||||
tools: list[Tool]
|
||||
"""The tools the model can call after the request's history: those loaded from
|
||||
the start and those the history has loaded since."""
|
||||
image_data: list[ImageData] = []
|
||||
"""The images the prompt's placeholders stand for, in order."""
|
||||
namespace_names: dict[str, tuple[str, str]] = {}
|
||||
"""The call name of each namespace member, mapped to its namespace and the
|
||||
name the request gives it there."""
|
||||
continued_turn: Optional[dict[str, Any]] = None
|
||||
"""The final assistant message the prompt continues, as a chat message."""
|
||||
|
||||
|
||||
class ChatOutputHooks(msgspec.Struct, frozen=True, kw_only=True):
|
||||
@@ -97,25 +78,6 @@ class ChatEncodingOutputAdapter(Protocol):
|
||||
) -> ResponsesOutputStream: ...
|
||||
|
||||
|
||||
class NativeRequestEncoder(Protocol):
|
||||
def encode_native(
|
||||
self,
|
||||
*,
|
||||
protocol: NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
thinking: bool,
|
||||
effort: Union[str, int, float, None],
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> NativePrompt:
|
||||
"""``body`` carries the request's conversation, tools and, for
|
||||
``tool_choice="none"``, its choice; ``thinking`` and ``effort`` are the
|
||||
handler's decisions, as for ``encode_messages``. ``response_schema`` is
|
||||
the JSON schema the answer follows; a trailing assistant turn is continued
|
||||
under ``continue_final_message`` and otherwise re-sent as a user turn."""
|
||||
...
|
||||
|
||||
|
||||
class ChatEncodingAdapter(Protocol):
|
||||
collects_openai_media: bool
|
||||
"""Whether media still has to be collected from the OpenAI messages after
|
||||
@@ -130,12 +92,6 @@ class ChatEncodingAdapter(Protocol):
|
||||
"""None when the configured parsers leave output to the text parsers."""
|
||||
...
|
||||
|
||||
@property
|
||||
def native_encoder(self) -> Optional[NativeRequestEncoder]:
|
||||
"""None when Responses and Messages requests that load tools mid-conversation
|
||||
have no rendering of their own wire form."""
|
||||
...
|
||||
|
||||
def encode_messages(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -1,105 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Compatibility shim for Dynamo's SGLang chat processor.
|
||||
|
||||
Upstream SGLang builds expose ``sglang.srt.entrypoints.openai.encoding_dsv4``
|
||||
with ``encode_messages(messages, *, thinking_mode, reasoning_effort) -> str``.
|
||||
Dynamo's frontend (``dynamo.frontend.sglang_prepost``) imports that module to
|
||||
render DeepSeek-V4/V4.1 prompts when the model ships no HF chat template.
|
||||
|
||||
Our DeepSeek-V4 fork replaced the encoder with the ``deepseek-recipe``
|
||||
renderer (``sglang.srt.parser.deepseek_recipe_renderer``). This module exposes
|
||||
the upstream API on top of the recipe, so ``--dyn-chat-processor sglang``
|
||||
works without installing a second SGLang build.
|
||||
|
||||
The returned string is tokenized by the caller. For the V4.1 checkpoint the
|
||||
resulting ids match ``DeepSeekRecipe.encode()`` (the native serving path)
|
||||
byte-for-byte; the renderer is the same code path, only the tokenizer call
|
||||
moves to the caller.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from typing import Any, Optional
|
||||
|
||||
from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
IMAGE_TOKEN,
|
||||
MODEL_NAMES,
|
||||
load_deepseek_recipe,
|
||||
pre_normalize,
|
||||
)
|
||||
|
||||
# The Dynamo frontend uses this encoder for the V4 family. Our deployment is
|
||||
# DeepSeek-V4.1-Flash; V4-Pro checkpoints need the ``dsv4`` spec instead.
|
||||
_CHAT_ENCODING_SPEC = "dsv41"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _renderer():
|
||||
recipe = load_deepseek_recipe()
|
||||
special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
|
||||
encoding = recipe.DeepseekV41Encoding().with_special_tokens(special_tokens)
|
||||
options = recipe.ConversionOptions().with_special_tokens(special_tokens)
|
||||
return recipe, special_tokens, encoding, options
|
||||
|
||||
|
||||
def encode_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
thinking_mode: str = "chat",
|
||||
reasoning_effort: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Render OpenAI-style ``messages`` to a DeepSeek-V4.1 prompt string.
|
||||
|
||||
Mirrors upstream ``encoding_dsv4.encode_messages``:
|
||||
|
||||
* ``thinking_mode`` is ``"thinking"`` or ``"chat"``;
|
||||
* ``reasoning_effort`` is a named tier (``"high"`` / ``"max"``) or None;
|
||||
* tool schemas are attached to the first system message by the caller
|
||||
(Dynamo's convention), not passed as a separate argument.
|
||||
|
||||
Raises ``ValueError`` for prompt shapes the recipe rejects.
|
||||
"""
|
||||
recipe, _special_tokens, encoding, options = _renderer()
|
||||
|
||||
encoding_messages = copy.deepcopy(messages)
|
||||
tools = None
|
||||
if encoding_messages:
|
||||
tools = encoding_messages[0].pop("tools", None)
|
||||
if not tools:
|
||||
tools = None
|
||||
|
||||
# Dynamo already flattens message content and re-serializes tool-call
|
||||
# arguments; keep this defensive normalization for direct callers.
|
||||
for message in encoding_messages:
|
||||
if message.get("content") is None:
|
||||
message["content"] = ""
|
||||
for call in message.get("tool_calls") or []:
|
||||
function = call.get("function") if isinstance(call, dict) else None
|
||||
if isinstance(function, dict) and isinstance(
|
||||
function.get("arguments"), (dict, list)
|
||||
):
|
||||
function["arguments"] = json.dumps(
|
||||
function["arguments"], ensure_ascii=False
|
||||
)
|
||||
|
||||
body: dict[str, Any] = {
|
||||
"messages": encoding_messages,
|
||||
"thinking": thinking_mode == "thinking",
|
||||
}
|
||||
if tools:
|
||||
body["tools"] = tools
|
||||
if reasoning_effort is not None:
|
||||
body["reasoning_effort"] = reasoning_effort
|
||||
|
||||
normalized = pre_normalize(body)
|
||||
normalized["model"] = MODEL_NAMES[_CHAT_ENCODING_SPEC]
|
||||
try:
|
||||
converted = recipe.ChatCompletionRequest(normalized).convert(options)
|
||||
except recipe.ConversionError as error: # type: ignore[attr-defined]
|
||||
detail = getattr(error, "body", None) or str(error)
|
||||
raise ValueError(f"DeepSeek-V4.1 prompt conversion failed: {detail}") from error
|
||||
|
||||
return encoding.render_conversation(converted.conversation).prompt
|
||||
@@ -17,7 +17,6 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
import msgspec
|
||||
import unicodedata
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
@@ -744,10 +743,6 @@ class ChatCompletionMessageGenericParam(BaseModel):
|
||||
name: Optional[str] = None
|
||||
phase: Optional[Literal["commentary", "final_answer"]] = None
|
||||
reasoning_content: Optional[str] = None
|
||||
action: Optional[str] = Field(
|
||||
default=None,
|
||||
description="Action tag for assistant messages (used by deepseek-recipe).",
|
||||
)
|
||||
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
|
||||
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
|
||||
# DeepSeek API prefix completion: continue this final assistant message.
|
||||
@@ -801,22 +796,14 @@ class Function(BaseModel):
|
||||
description: Optional[str] = Field(default=None, examples=[None])
|
||||
name: str
|
||||
parameters: Optional[object] = None
|
||||
strict: Optional[bool] = None
|
||||
strict: bool = False
|
||||
defer_loading: Optional[bool] = None
|
||||
namespace: Optional[str] = Field(
|
||||
default=None,
|
||||
description="Tool namespace for disambiguation (used by deepseek-recipe).",
|
||||
)
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
def _serialize(self, handler):
|
||||
data = handler(self)
|
||||
if self.defer_loading is None:
|
||||
data.pop("defer_loading", None)
|
||||
if self.strict is None:
|
||||
data.pop("strict", None)
|
||||
if self.namespace is None:
|
||||
data.pop("namespace", None)
|
||||
return data
|
||||
|
||||
|
||||
@@ -834,26 +821,6 @@ class Tool(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class ChatCompletionNamespaceTool(BaseModel):
|
||||
"""A Responses-style namespace grouping sent to the Chat Completions API.
|
||||
|
||||
Chat Completions has no namespace field in its tool-call shape, so the
|
||||
members are flattened to ``<namespace>__<function>`` before rendering; the
|
||||
model calls that flat name and the response reports it back unchanged.
|
||||
"""
|
||||
|
||||
type: Literal["namespace"]
|
||||
name: str
|
||||
description: Optional[str] = None
|
||||
tools: List[Tool]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_members(self) -> "ChatCompletionNamespaceTool":
|
||||
if not self.tools:
|
||||
raise ValueError("namespace tools must list at least one function tool")
|
||||
return self
|
||||
|
||||
|
||||
# Tool is defined after the message params that reference it, so the forward
|
||||
# reference has to be resolved explicitly.
|
||||
ChatCompletionMessageGenericParam.model_rebuild()
|
||||
@@ -941,22 +908,11 @@ class ChatCompletionRequest(PDRoutingFields):
|
||||
temperature: Optional[float] = None
|
||||
top_p: Optional[float] = None
|
||||
user: Optional[str] = None
|
||||
tools: Optional[List[Union[Tool, ChatCompletionNamespaceTool]]] = Field(
|
||||
default=None, examples=[None]
|
||||
)
|
||||
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
|
||||
# Set by /v1/responses when the request has namespace tools: the chat template
|
||||
# renders these, while ``tools`` (flattened to ``<namespace>.<function>``) still
|
||||
# drives tool-call parsing.
|
||||
_template_tools: Optional[List[Dict[str, Any]]] = PrivateAttr(default=None)
|
||||
# Set with ``input_ids`` by a caller that rendered them itself: the images the
|
||||
# prompt's placeholders stand for, in order.
|
||||
_input_image_data: Optional[List[Any]] = PrivateAttr(default=None)
|
||||
# Whether ``tools`` are the only tools the rendered prompt shows, so a call
|
||||
# to any other tool is dropped rather than returned.
|
||||
_drops_unlisted_tool_calls: bool = PrivateAttr(default=False)
|
||||
# Set with ``input_ids`` under ``continue_final_message``: the final assistant
|
||||
# message the prompt continues, where output parsing resumes.
|
||||
_continued_turn: Optional[Dict[str, Any]] = PrivateAttr(default=None)
|
||||
tool_choice: Union[ToolChoice, Literal["auto", "required", "none"]] = Field(
|
||||
default="auto", examples=["none"]
|
||||
) # noqa
|
||||
@@ -1184,39 +1140,6 @@ class ChatCompletionRequest(PDRoutingFields):
|
||||
|
||||
return values
|
||||
|
||||
@model_validator(mode="after")
|
||||
def flatten_namespace_tools(self) -> "ChatCompletionRequest":
|
||||
"""Render namespace groups as flat ``ns__function`` chat tools."""
|
||||
if not self.tools:
|
||||
return self
|
||||
flattened: List[Tool] = []
|
||||
for tool in self.tools:
|
||||
if isinstance(tool, ChatCompletionNamespaceTool):
|
||||
for member in tool.tools:
|
||||
name = f"{tool.name}__{member.function.name}"
|
||||
description = member.function.description
|
||||
if tool.description:
|
||||
description = (
|
||||
tool.description + chr(10) + description
|
||||
if description
|
||||
else tool.description
|
||||
)
|
||||
flattened.append(
|
||||
Tool(
|
||||
type="function",
|
||||
function=Function(
|
||||
name=name,
|
||||
description=description,
|
||||
parameters=member.function.parameters,
|
||||
strict=member.function.strict,
|
||||
),
|
||||
)
|
||||
)
|
||||
else:
|
||||
flattened.append(tool)
|
||||
self.tools = flattened
|
||||
return self
|
||||
|
||||
def to_sampling_params(
|
||||
self,
|
||||
stop: List[str],
|
||||
@@ -1329,7 +1252,6 @@ class ChatMessage(BaseModel):
|
||||
role: Optional[str] = None
|
||||
content: Optional[str] = None
|
||||
reasoning_content: Optional[str] = None
|
||||
action: Optional[str] = None
|
||||
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
@@ -1337,8 +1259,6 @@ class ChatMessage(BaseModel):
|
||||
data = handler(self)
|
||||
if self.tool_calls is None:
|
||||
data.pop("tool_calls", None)
|
||||
if self.action is None:
|
||||
data.pop("action", None)
|
||||
return data
|
||||
|
||||
|
||||
@@ -1891,8 +1811,6 @@ RESPONSE_TOOL_TYPES = Literal[
|
||||
class ResponseTool(BaseModel):
|
||||
"""Tool definition for responses."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
type: RESPONSE_TOOL_TYPES = Field(description="Type of tool to enable")
|
||||
name: Optional[str] = None
|
||||
description: Optional[str] = None
|
||||
@@ -1912,40 +1830,6 @@ class ResponseTool(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class ResponseToolState(msgspec.Struct, frozen=True):
|
||||
"""Public tool metadata and name mappings for one translated request."""
|
||||
|
||||
public_tools: list[ResponseTool] | None = None
|
||||
public_choice: str | dict[str, Any] | None = None
|
||||
names: dict[str, dict[str, str]] = {}
|
||||
# The tools the model can call after the history, from a request rendered in
|
||||
# its own wire form; None when `tools` alone declares them.
|
||||
chat_tools: list[Tool] | None = None
|
||||
# ``custom`` tools a ``tool_search_output`` loaded, beside the declared ones.
|
||||
discovered_custom_names: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
class ResponseToolSearchCall(BaseModel):
|
||||
"""A client-executed ``tool_search`` call; the pinned openai SDK has no type."""
|
||||
|
||||
id: str
|
||||
type: Literal["tool_search_call"] = "tool_search_call"
|
||||
call_id: str
|
||||
execution: Literal["client"] = "client"
|
||||
arguments: Dict[str, Any]
|
||||
status: Literal["in_progress", "completed", "incomplete"]
|
||||
|
||||
|
||||
class ResponseToolSearchCallEvent(BaseModel):
|
||||
"""``response.output_item.added`` / ``.done`` for a tool search call, which the
|
||||
SDK's event types reject."""
|
||||
|
||||
type: Literal["response.output_item.added", "response.output_item.done"]
|
||||
sequence_number: int
|
||||
output_index: int
|
||||
item: ResponseToolSearchCall
|
||||
|
||||
|
||||
class ResponseInputMessageParam(EasyInputMessageParam, total=False):
|
||||
phase: Optional[Literal["commentary", "final_answer"]]
|
||||
|
||||
@@ -1972,10 +1856,6 @@ ResponseInputOutputItem: TypeAlias = Union[
|
||||
class ResponsesRequest(PDRoutingFields):
|
||||
"""Request body for v1/responses endpoint."""
|
||||
|
||||
_tool_state: ResponseToolState = PrivateAttr(default_factory=ResponseToolState)
|
||||
# By call_id, the name the model called a tool by when the output item
|
||||
# reports another; the stored history replays the call under it.
|
||||
_called_tool_names: Dict[str, str] = PrivateAttr(default_factory=dict)
|
||||
_response_parser_prefix: str = PrivateAttr(default="")
|
||||
_response_parser_grammar_start: Optional[str] = PrivateAttr(default=None)
|
||||
_response_parser_json_response_format: bool = PrivateAttr(default=False)
|
||||
@@ -2172,10 +2052,7 @@ class ResponsesRequest(PDRoutingFields):
|
||||
"name"
|
||||
)
|
||||
if tool_choice.get("type") in ("function", "custom") and name:
|
||||
choice = {"type": tool_choice["type"], "name": name}
|
||||
if "namespace" in tool_choice:
|
||||
choice["namespace"] = tool_choice["namespace"]
|
||||
return choice
|
||||
return {"type": "function", "name": name}
|
||||
return "auto"
|
||||
|
||||
def to_sampling_params(
|
||||
@@ -2299,7 +2176,6 @@ class ResponsesResponse(BaseModel):
|
||||
ResponseOutputItem,
|
||||
ResponseReasoningItem,
|
||||
ResponseFunctionToolCall,
|
||||
ResponseToolSearchCall,
|
||||
]
|
||||
] = Field(default_factory=list)
|
||||
status: Literal[
|
||||
@@ -2365,7 +2241,6 @@ class ResponsesResponse(BaseModel):
|
||||
ResponseOutputItem,
|
||||
ResponseReasoningItem,
|
||||
ResponseFunctionToolCall,
|
||||
ResponseToolSearchCall,
|
||||
]
|
||||
],
|
||||
status: str,
|
||||
|
||||
@@ -957,13 +957,12 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
The reasoning parser is told the same thing, so channel-framed formats
|
||||
keep their framing intact exactly when a tool-call parser consumes it.
|
||||
"""
|
||||
if not self.tool_call_parser or request._response_parser_json_response_format:
|
||||
return False
|
||||
if request._drops_unlisted_tool_calls and request.tool_choice == "none":
|
||||
# The history showed the model tools, so it can still write a call;
|
||||
# parsing drops it, where skipping the parser would return its markup.
|
||||
return True
|
||||
return bool(request.tool_choice != "none" and self._effective_tools(request))
|
||||
return bool(
|
||||
request.tool_choice != "none"
|
||||
and self._effective_tools(request)
|
||||
and self.tool_call_parser
|
||||
and not request._response_parser_json_response_format
|
||||
)
|
||||
|
||||
def _validate_request(self, request: ChatCompletionRequest) -> str | None:
|
||||
"""Validate that the input is valid."""
|
||||
@@ -1368,7 +1367,18 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
self, request: ChatCompletionRequest, is_multimodal: bool
|
||||
) -> MessageProcessingResult:
|
||||
"""Process chat messages and apply chat template"""
|
||||
self._apply_request_defaults(request)
|
||||
if self.default_chat_template_kwargs:
|
||||
ctk = dict(request.chat_template_kwargs or {})
|
||||
for k, v in self.default_chat_template_kwargs.items():
|
||||
ctk.setdefault(k, v)
|
||||
request.chat_template_kwargs = ctk
|
||||
effort = ctk.get("reasoning_effort")
|
||||
if effort is not None and request.reasoning_effort is None:
|
||||
request.reasoning_effort = effort
|
||||
|
||||
normalize_hunyuan_reasoning_effort(
|
||||
request, self.reasoning_parser, self.template_manager.reasoning_config
|
||||
)
|
||||
|
||||
# GptOss model needs to keep special tokens for harmony parsing
|
||||
if self.is_gpt_oss or self.is_gemma4:
|
||||
@@ -1474,7 +1484,7 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
result = MessageProcessingResult(
|
||||
prompt="",
|
||||
prompt_ids=request.input_ids,
|
||||
image_data=request._input_image_data,
|
||||
image_data=None,
|
||||
audio_data=None,
|
||||
video_data=None,
|
||||
modalities=[],
|
||||
@@ -1530,64 +1540,6 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
result.reasoning_end_token_ids = list(token_ids)
|
||||
return result
|
||||
|
||||
def _apply_request_defaults(self, request: ChatCompletionRequest) -> None:
|
||||
"""Settle the thinking and effort that rendering reads; repeating it
|
||||
changes nothing."""
|
||||
if self.default_chat_template_kwargs:
|
||||
ctk = dict(request.chat_template_kwargs or {})
|
||||
for k, v in self.default_chat_template_kwargs.items():
|
||||
ctk.setdefault(k, v)
|
||||
request.chat_template_kwargs = ctk
|
||||
effort = ctk.get("reasoning_effort")
|
||||
if effort is not None and request.reasoning_effort is None:
|
||||
request.reasoning_effort = effort
|
||||
|
||||
normalize_hunyuan_reasoning_effort(
|
||||
request, self.reasoning_parser, self.template_manager.reasoning_config
|
||||
)
|
||||
|
||||
@property
|
||||
def native_tool_loading(self) -> bool:
|
||||
"""Whether Responses and Messages requests that load tools mid-conversation
|
||||
render from their own wire form (see `prepare_native_prompt`)."""
|
||||
return (
|
||||
self._encoding_adapter is not None
|
||||
and self._encoding_adapter.native_encoder is not None
|
||||
)
|
||||
|
||||
def prepare_native_prompt(
|
||||
self,
|
||||
request: ChatCompletionRequest,
|
||||
*,
|
||||
protocol: chat_encoding_adapters.NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
) -> chat_encoding_adapters.NativePrompt:
|
||||
"""Render `body`, the Responses or Messages form of `request`, into its
|
||||
`input_ids` and images, and set its `tools` to the tools the model can call
|
||||
after the history; rendering then keeps only grammar, stops and parsing."""
|
||||
self._apply_request_defaults(request)
|
||||
response_format = request.response_format
|
||||
prompt = self._encoding_adapter.native_encoder.encode_native(
|
||||
protocol=protocol,
|
||||
body=body,
|
||||
thinking=self._get_reasoning_from_request(request),
|
||||
effort=request.reasoning_effort,
|
||||
response_schema=(
|
||||
response_format.json_schema.schema_
|
||||
if response_format is not None and response_format.json_schema
|
||||
else None
|
||||
),
|
||||
continue_final_message=request.continue_final_message,
|
||||
)
|
||||
if prompt.image_data and not self.tokenizer_manager.model_config.is_multimodal:
|
||||
raise ValueError("image input is not supported for this model")
|
||||
request.input_ids = prompt.prompt_ids
|
||||
request.tools = prompt.tools or None
|
||||
request._input_image_data = prompt.image_data or None
|
||||
request._drops_unlisted_tool_calls = True
|
||||
request._continued_turn = prompt.continued_turn
|
||||
return prompt
|
||||
|
||||
def _apply_jinja_template(
|
||||
self,
|
||||
request: ChatCompletionRequest,
|
||||
|
||||
@@ -5,20 +5,15 @@ request semantics."""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any, Optional, Union
|
||||
|
||||
from sglang.srt.entrypoints.openai.chat_encoding_adapters import (
|
||||
ADAPTER_OUTPUT_PARSERS,
|
||||
NativePrompt,
|
||||
NativeProtocol,
|
||||
select_new_output_ids,
|
||||
token_output_sse_chunks,
|
||||
)
|
||||
from sglang.srt.entrypoints.openai.protocol import (
|
||||
Function,
|
||||
FunctionResponse,
|
||||
Tool,
|
||||
ToolCall,
|
||||
)
|
||||
from sglang.srt.environ import envs
|
||||
@@ -31,9 +26,7 @@ from sglang.srt.parser.deepseek_recipe_output import (
|
||||
whole_tool_calls,
|
||||
)
|
||||
from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
BUDGET_TIER,
|
||||
DeepSeekRecipe,
|
||||
loaded_tools,
|
||||
parse_effort_budget,
|
||||
)
|
||||
from sglang.srt.runtime_context import get_serving
|
||||
@@ -44,6 +37,7 @@ if TYPE_CHECKING:
|
||||
from sglang.srt.entrypoints.openai.protocol import (
|
||||
ChatCompletionRequest,
|
||||
ResponsesRequest,
|
||||
Tool,
|
||||
)
|
||||
|
||||
DEFAULT_EFFORT_ENVS = {
|
||||
@@ -51,12 +45,6 @@ DEFAULT_EFFORT_ENVS = {
|
||||
"dsv41": envs.SGLANG_DSV41_REASONING_EFFORT,
|
||||
}
|
||||
JSON_RESPONSE_FORMATS = ("json_object", "json_schema")
|
||||
# V4 renders deferred tools up front and never loads one mid-conversation.
|
||||
NATIVE_TOOL_LOADING_SPEC = "dsv41"
|
||||
NATIVE_EFFORT_FIELDS: dict[str, str] = {
|
||||
"responses": "reasoning",
|
||||
"messages": "output_config",
|
||||
}
|
||||
|
||||
|
||||
class DeepSeekRecipeChatEncodingAdapter:
|
||||
@@ -108,13 +96,6 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
def output(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
|
||||
return self if self._token_output else None
|
||||
|
||||
def validate_request(self, request: ChatCompletionRequest) -> Optional[str]:
|
||||
return None
|
||||
|
||||
@property
|
||||
def native_encoder(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
|
||||
return self if self._spec == NATIVE_TOOL_LOADING_SPEC else None
|
||||
|
||||
def encode_messages(
|
||||
self,
|
||||
*,
|
||||
@@ -168,66 +149,6 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
prompt_ids += self.deepseek_recipe.encode_text(task_prefix)
|
||||
return prompt_ids
|
||||
|
||||
def encode_native(
|
||||
self,
|
||||
*,
|
||||
protocol: NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
thinking: bool,
|
||||
effort: Union[str, int, float, None],
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> NativePrompt:
|
||||
tier, budget = self._effort(effort)
|
||||
# recipe reads an effort as a request to think.
|
||||
if thinking and tier is not None:
|
||||
body = {**body, NATIVE_EFFORT_FIELDS[protocol]: {"effort": tier}}
|
||||
conversation = self.deepseek_recipe.with_host_settings(
|
||||
self.deepseek_recipe.convert_native(protocol, body, thinking=thinking),
|
||||
response_schema=response_schema,
|
||||
continue_final_message=continue_final_message,
|
||||
)
|
||||
last = conversation.messages[-1] if conversation.messages else None
|
||||
# recipe's Messages conversion drops `strict`, which SGLang's grammar enforces.
|
||||
strict = {tool.get("name") for tool in body["tools"] if tool.get("strict")}
|
||||
tools = [
|
||||
Tool(
|
||||
type="function",
|
||||
function=Function(
|
||||
name=tool.name,
|
||||
description=tool.description,
|
||||
parameters=tool.parameters,
|
||||
strict=True if tool.name in strict else tool.strict,
|
||||
),
|
||||
)
|
||||
for tool in loaded_tools(conversation)
|
||||
]
|
||||
return NativePrompt(
|
||||
prompt_ids=self.deepseek_recipe.encode_conversation(
|
||||
conversation, effort_budget=budget
|
||||
),
|
||||
tools=tools,
|
||||
image_data=[
|
||||
ImageData(url=url)
|
||||
for url in self.deepseek_recipe.image_urls(conversation)
|
||||
],
|
||||
namespace_names=(
|
||||
self.deepseek_recipe.namespace_tool_names(_responses_namespaces(body))
|
||||
if protocol == "responses"
|
||||
else {}
|
||||
),
|
||||
continued_turn=(
|
||||
{"role": "assistant", "content": last.content}
|
||||
| (
|
||||
{"reasoning_content": last.reasoning_content}
|
||||
if last.reasoning_content is not None
|
||||
else {}
|
||||
)
|
||||
if last is not None and last.prefix
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _encode(
|
||||
self,
|
||||
body: dict[str, Any],
|
||||
@@ -236,29 +157,21 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
task: Optional[str] = None,
|
||||
image_urls: Optional[list[str]] = None,
|
||||
) -> list[int]:
|
||||
tier, budget = self._effort(effort)
|
||||
if budget is None and tier is not None:
|
||||
body["reasoning_effort"] = tier
|
||||
if effort is None or effort == "none":
|
||||
effort = self._default_effort
|
||||
budget = None
|
||||
if effort is not None and not isinstance(effort, str):
|
||||
if self._spec != "dsv41":
|
||||
raise ValueError(
|
||||
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
|
||||
)
|
||||
budget = parse_effort_budget(effort)
|
||||
elif effort is not None:
|
||||
body["reasoning_effort"] = effort
|
||||
return self.deepseek_recipe.encode(
|
||||
body, task=task, effort_budget=budget, image_urls=image_urls
|
||||
)
|
||||
|
||||
def _effort(
|
||||
self, effort: Union[str, int, float, None]
|
||||
) -> tuple[Optional[str], Optional[int]]:
|
||||
"""The named tier recipe renders and, for a V4.1 integer budget, the budget
|
||||
that replaces the tier's line."""
|
||||
effort = _numeric_effort(effort)
|
||||
if effort is None or effort == "none":
|
||||
effort = self._default_effort
|
||||
if effort is None or isinstance(effort, str):
|
||||
return effort, None
|
||||
if self._spec != "dsv41":
|
||||
raise ValueError(
|
||||
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
|
||||
)
|
||||
return BUDGET_TIER, parse_effort_budget(effort)
|
||||
|
||||
def parse_chat_response(
|
||||
self,
|
||||
*,
|
||||
@@ -407,7 +320,7 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
self, request: ChatCompletionRequest
|
||||
) -> DeepSeekRecipeOutputParser:
|
||||
response_format = request.response_format
|
||||
continued = request._continued_turn or (
|
||||
continued = (
|
||||
_trailing_assistant_turn(
|
||||
[
|
||||
message.model_dump(exclude_unset=True, by_alias=True, mode="json")
|
||||
@@ -431,11 +344,6 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
separate_reasoning=self._separate_reasoning and request.separate_reasoning,
|
||||
stream_reasoning=request.stream_reasoning,
|
||||
stop=_stop_strings(request.stop),
|
||||
callable_names=(
|
||||
_native_callable_names(request.tools or [], request.tool_choice)
|
||||
if request._drops_unlisted_tool_calls
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _new_responses_parser(
|
||||
@@ -445,25 +353,12 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
separate_reasoning: bool,
|
||||
parse_tool_calls: bool,
|
||||
) -> DeepSeekRecipeOutputParser:
|
||||
native = request._tool_state.chat_tools is not None
|
||||
return DeepSeekRecipeOutputParser(
|
||||
self.deepseek_recipe,
|
||||
thinking_stage=self.deepseek_recipe.parse_start(thinking=require_reasoning),
|
||||
parse_tool_calls=parse_tool_calls,
|
||||
# Under `none` the tool parser runs only to drop calls; a JSON answer
|
||||
# must be read as JSON, or a call marker spelled in a string ends it.
|
||||
parse_json_output=native
|
||||
and request.tool_choice == "none"
|
||||
and request.has_json_schema_constraint(),
|
||||
separate_reasoning=separate_reasoning,
|
||||
stop=_stop_strings(request.stop),
|
||||
callable_names=(
|
||||
_native_callable_names(
|
||||
request._tool_state.chat_tools, request.tool_choice
|
||||
)
|
||||
if native
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _tool_call_id(self, index: int, name: str, history_tool_calls_cnt: int) -> str:
|
||||
@@ -495,37 +390,6 @@ class DeepSeekRecipeResponsesStream:
|
||||
return delta.reasoning, delta.content, _tool_call_items(delta.tool_calls)
|
||||
|
||||
|
||||
def _responses_namespaces(body: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""The namespace tools a Responses body declares and its history discovers, in
|
||||
the order recipe's conversion names their members."""
|
||||
discovered = [
|
||||
tool
|
||||
for item in body["input"]
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
for tool in item["tools"]
|
||||
]
|
||||
return [
|
||||
tool for tool in [*body["tools"], *discovered] if tool["type"] == "namespace"
|
||||
]
|
||||
|
||||
|
||||
def _null_content_as_empty(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""As `_process_messages` renders them: a null content is ""."""
|
||||
return [
|
||||
message if message.get("content") is not None else {**message, "content": ""}
|
||||
for message in messages
|
||||
]
|
||||
|
||||
|
||||
def _numeric_effort(
|
||||
effort: Union[str, int, float, None],
|
||||
) -> Union[str, int, float, None]:
|
||||
"""A numeric string effort ("30", "0.3") as the number it spells."""
|
||||
if isinstance(effort, str) and re.fullmatch(r"\d+(\.\d+)?", effort.strip()):
|
||||
return float(effort) if "." in effort else int(effort)
|
||||
return effort
|
||||
|
||||
|
||||
def _trailing_assistant_turn(
|
||||
messages: list[dict[str, Any]],
|
||||
) -> Optional[dict[str, Any]]:
|
||||
@@ -541,18 +405,6 @@ def _stop_strings(stop: Union[str, list[str], None]) -> list[str]:
|
||||
return [stop] if isinstance(stop, str) else stop or []
|
||||
|
||||
|
||||
def _native_callable_names(
|
||||
tools: list[Tool], tool_choice: Any
|
||||
) -> Optional[frozenset[str]]:
|
||||
"""The names of `tools`, all that a natively rendered prompt shows the model;
|
||||
None when calls to other tools are returned. `none` allows no call at all."""
|
||||
if tool_choice == "none":
|
||||
return frozenset()
|
||||
if envs.SGLANG_FORWARD_UNKNOWN_TOOLS.get():
|
||||
return None
|
||||
return frozenset(tool.function.name for tool in tools)
|
||||
|
||||
|
||||
def _tool_call_items(
|
||||
calls: Sequence[Union[RecipeToolCall, RecipeToolCallDelta]],
|
||||
) -> list[ToolCallItem]:
|
||||
|
||||
@@ -36,10 +36,7 @@ if TYPE_CHECKING:
|
||||
|
||||
class InklingChatEncodingAdapter:
|
||||
collects_openai_media = True
|
||||
history_tool_call_arguments = "strict"
|
||||
renders_response_format = False
|
||||
uncounted_prompt_tokens = 0
|
||||
native_encoder = None
|
||||
strict_tool_call_arguments = True
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
||||
@@ -13,7 +13,6 @@ from http import HTTPStatus
|
||||
from typing import TYPE_CHECKING, Any, AsyncGenerator, AsyncIterator, Optional, Union
|
||||
|
||||
import jinja2
|
||||
import msgspec
|
||||
import openai.types.responses as openai_responses_types
|
||||
import orjson
|
||||
from fastapi import Request
|
||||
@@ -44,7 +43,6 @@ from openai.types.responses.response_reasoning_summary_part_done_event import (
|
||||
Part as ResponseReasoningSummaryDonePart,
|
||||
)
|
||||
from openai_harmony import Message as OpenAIMessage
|
||||
from pydantic import BaseModel, TypeAdapter
|
||||
|
||||
from sglang.srt.entrypoints.context import (
|
||||
ConversationContext,
|
||||
@@ -72,12 +70,7 @@ from sglang.srt.entrypoints.openai.protocol import (
|
||||
ResponseOutputMessage,
|
||||
ResponsesRequest,
|
||||
ResponsesResponse,
|
||||
ResponseTool,
|
||||
ResponseToolSearchCall,
|
||||
ResponseToolSearchCallEvent,
|
||||
Tool,
|
||||
ToolChoice,
|
||||
ToolChoiceFuncName,
|
||||
UsageInfo,
|
||||
)
|
||||
from sglang.srt.entrypoints.openai.responses_adapters import (
|
||||
@@ -121,100 +114,6 @@ class _MediaInputValidationError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class ResponseToolChoiceError(ValueError):
|
||||
"""A response tool_choice that names a tool the model cannot call."""
|
||||
|
||||
|
||||
TOOL_SEARCH = "tool_search"
|
||||
TOOL_SEARCH_ITEMS = ("tool_search_call", "tool_search_output")
|
||||
APPLY_PATCH = "apply_patch"
|
||||
NAMESPACE_MEMBERS_ERROR = (
|
||||
"namespace tools must list function tools, each with a type and a name"
|
||||
)
|
||||
|
||||
|
||||
def _holds_function_tools(members: Any) -> bool:
|
||||
return isinstance(members, list) and all(
|
||||
isinstance(member, dict)
|
||||
and member.get("type") == "function"
|
||||
and isinstance(member.get("name"), str)
|
||||
for member in members
|
||||
)
|
||||
|
||||
|
||||
def _defers_loading(tool: ResponseTool) -> bool:
|
||||
if tool.type == "namespace":
|
||||
return bool(tool.defer_loading) or any(
|
||||
isinstance(member, dict) and member.get("defer_loading")
|
||||
for member in tool.tools or []
|
||||
)
|
||||
return tool.type in ("function", "custom") and bool(tool.defer_loading)
|
||||
|
||||
|
||||
def _defer_tool_loading(
|
||||
tools: list[dict[str, Any]], items: list
|
||||
) -> tuple[list[dict[str, Any]], list]:
|
||||
"""`tools` without those that defer loading, and `items` whose
|
||||
`tool_search_output` entries naming a deferred tool take the fields they
|
||||
leave out from its declaration, so a search result loads it where it returns.
|
||||
recipe's Responses conversion renders a deferred tool up front and rejects a
|
||||
search result that repeats a declared name."""
|
||||
deferred: dict[tuple[Optional[str], str], dict[str, Any]] = {}
|
||||
shown = []
|
||||
for tool in tools:
|
||||
if tool["type"] == "namespace":
|
||||
members = []
|
||||
for member in tool["tools"]:
|
||||
if tool.get("defer_loading") or member.get("defer_loading"):
|
||||
deferred[(tool["name"], member["name"])] = member
|
||||
else:
|
||||
members.append(member)
|
||||
if members:
|
||||
shown.append({**tool, "tools": members})
|
||||
elif tool["type"] in ("function", "custom") and tool.get("defer_loading"):
|
||||
deferred[(None, tool["name"])] = tool
|
||||
else:
|
||||
shown.append(tool)
|
||||
|
||||
def loaded(tool: dict[str, Any], namespace: Optional[str] = None) -> dict:
|
||||
if tool["type"] == "namespace":
|
||||
members = [loaded(member, tool["name"]) for member in tool["tools"]]
|
||||
return {**tool, "tools": members}
|
||||
declared = deferred.get((namespace, tool.get("name")))
|
||||
if declared is None or declared["type"] != tool["type"]:
|
||||
return tool
|
||||
return declared | {k: v for k, v in tool.items() if v is not None}
|
||||
|
||||
if deferred:
|
||||
items = [
|
||||
(
|
||||
{**item, "tools": [loaded(tool) for tool in item["tools"]]}
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
else item
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
return shown, items
|
||||
|
||||
|
||||
def _tool_search_call(
|
||||
item_id: str, call_id: str, arguments: str
|
||||
) -> ResponseToolSearchCall:
|
||||
"""A search call reports object arguments; one whose arguments are not a JSON
|
||||
object is incomplete."""
|
||||
try:
|
||||
parsed = json.loads(arguments)
|
||||
except ValueError:
|
||||
parsed = None
|
||||
complete = isinstance(parsed, dict)
|
||||
return ResponseToolSearchCall(
|
||||
id=item_id,
|
||||
call_id=call_id,
|
||||
arguments=parsed if complete else {},
|
||||
status="completed" if complete else "incomplete",
|
||||
)
|
||||
|
||||
|
||||
def _build_output_text_logprobs(meta_info: dict) -> list[Logprob]:
|
||||
"""Reshape decoded ``meta_info`` logprobs into the Responses logprob type,
|
||||
covering every generated token."""
|
||||
@@ -420,21 +319,9 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
# FIXME: If the engine is dead, raise an error
|
||||
# This is required for the streaming case
|
||||
|
||||
# Such a request's callable tools are known once its history renders,
|
||||
# so `_make_request` checks its tool_choice instead.
|
||||
native = self._renders_natively(request)
|
||||
if native:
|
||||
tools_error = self._native_tools_error(request)
|
||||
if tools_error is not None:
|
||||
return self.create_error_response(tools_error, param="tools")
|
||||
|
||||
# ``tool_choice="required"`` needs a tool the parser can actually force.
|
||||
if (
|
||||
not native
|
||||
and request.tool_choice == "required"
|
||||
and not any(
|
||||
tool.type in ("function", "custom") for tool in (request.tools or [])
|
||||
)
|
||||
if request.tool_choice == "required" and not any(
|
||||
tool.type in ("function", "custom") for tool in (request.tools or [])
|
||||
):
|
||||
return self.create_error_response(
|
||||
'tool_choice="required" requires at least one tool with '
|
||||
@@ -443,13 +330,9 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
|
||||
tool_choice = request.effective_tool_choice()
|
||||
if (
|
||||
not native
|
||||
and isinstance(tool_choice, dict)
|
||||
and not any(
|
||||
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
|
||||
for tool in request.tools or []
|
||||
)
|
||||
if isinstance(tool_choice, dict) and not any(
|
||||
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
|
||||
for tool in request.tools or []
|
||||
):
|
||||
return self.create_error_response(
|
||||
f"Tool {tool_choice['name']!r} is not declared in tools",
|
||||
@@ -532,8 +415,6 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
|
||||
except _MediaInputValidationError as e:
|
||||
return self.create_error_response(str(e))
|
||||
except ResponseToolChoiceError as e:
|
||||
return self.create_error_response(str(e), param="tool_choice")
|
||||
except (ValueError, TypeError, RuntimeError, jinja2.TemplateError) as e:
|
||||
logger.exception("Error in preprocessing prompt inputs")
|
||||
return self.create_error_response(f"{e} {e.__cause__}")
|
||||
@@ -803,10 +684,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
prev_response: Optional[ResponsesResponse],
|
||||
tokenizer: Any,
|
||||
):
|
||||
native = self._renders_natively(request)
|
||||
messages = (
|
||||
[] if native else self._construct_input_messages(request, prev_response)
|
||||
)
|
||||
messages = self._construct_input_messages(request, prev_response)
|
||||
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
chat_request = ChatCompletionRequest(
|
||||
@@ -829,8 +707,6 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
chat_template_kwargs=request.chat_template_kwargs,
|
||||
)
|
||||
chat_request._template_tools = self._response_tools_to_template_tools(request)
|
||||
if native:
|
||||
self._prepare_native_responses_prompt(request, chat_request)
|
||||
|
||||
media_error = self._validate_media_content(chat_request)
|
||||
if media_error:
|
||||
@@ -1030,7 +906,10 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
else:
|
||||
self.msg_store[response.id] = self._response_input_history(
|
||||
request
|
||||
) + self._replayed_output_items(request, response.output)
|
||||
) + [
|
||||
item.model_dump(exclude_none=True)
|
||||
for item in response.output
|
||||
]
|
||||
|
||||
return response
|
||||
|
||||
@@ -1073,18 +952,12 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
name: str,
|
||||
arguments: str,
|
||||
custom_names: set[str],
|
||||
tool_names: dict[str, dict[str, str]],
|
||||
*,
|
||||
client_tool_search: bool = False,
|
||||
) -> Union[
|
||||
ResponseFunctionToolCall, ResponseCustomToolCall, ResponseToolSearchCall
|
||||
]:
|
||||
namespace_names: Optional[dict[str, tuple[str, str]]] = None,
|
||||
) -> Union[ResponseFunctionToolCall, ResponseCustomToolCall]:
|
||||
"""A call against a ``custom`` tool reports its freeform payload rather
|
||||
than the JSON arguments of the shim function tool. A call against a
|
||||
namespace member reports ``namespace`` and the bare function ``name``."""
|
||||
call_id = f"call_{random_uuid()[:24]}"
|
||||
if client_tool_search and name == TOOL_SEARCH:
|
||||
return _tool_search_call(f"tsc_{random_uuid()[:8]}", call_id, arguments)
|
||||
if name in custom_names:
|
||||
return ResponseCustomToolCall(
|
||||
type="custom_tool_call",
|
||||
@@ -1093,13 +966,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
name=name,
|
||||
input=decode_custom_tool_input(arguments),
|
||||
)
|
||||
entry = (tool_names or {}).get(name)
|
||||
if isinstance(entry, tuple):
|
||||
namespace, name = entry
|
||||
elif isinstance(entry, dict):
|
||||
namespace, name = entry.get("namespace"), entry.get("name", name)
|
||||
else:
|
||||
namespace = None
|
||||
namespace, name = (namespace_names or {}).get(name, (None, name))
|
||||
return ResponseFunctionToolCall(
|
||||
arguments=arguments,
|
||||
call_id=call_id,
|
||||
@@ -1196,17 +1063,12 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
finish_reason: Optional[dict[str, Any]] = None,
|
||||
):
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
custom_names = self._custom_tool_names(request)
|
||||
custom_names = custom_tool_names(request.tools)
|
||||
namespace_names = (
|
||||
namespace_function_names(request.tools)
|
||||
if self.tool_call_parser == "muse"
|
||||
else {
|
||||
call_name: (entry.get("namespace"), entry.get("name", call_name))
|
||||
for call_name, entry in request._tool_state.names.items()
|
||||
if entry.get("namespace")
|
||||
}
|
||||
else {}
|
||||
)
|
||||
client_tool_search = self._client_tool_search(request)
|
||||
encoding_output = self._encoding_output()
|
||||
if encoding_output is not None:
|
||||
required_json = self._uses_required_tool_json(request, chat_tools)
|
||||
@@ -1226,17 +1088,12 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
content, custom_names, namespace_names
|
||||
)
|
||||
else:
|
||||
tool_call_items = []
|
||||
for call in calls:
|
||||
item = self._make_tool_call_item(
|
||||
call.name,
|
||||
call.parameters,
|
||||
custom_names,
|
||||
namespace_names,
|
||||
client_tool_search=client_tool_search,
|
||||
tool_call_items = [
|
||||
self._make_tool_call_item(
|
||||
call.name, call.parameters, custom_names, namespace_names
|
||||
)
|
||||
self._record_called_name(request, item.call_id, call.name)
|
||||
tool_call_items.append(item)
|
||||
for call in calls
|
||||
]
|
||||
else:
|
||||
reasoning_content, content, tool_call_items = self._parse_text_output(
|
||||
request,
|
||||
@@ -1337,15 +1194,14 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
try:
|
||||
content, call_info_list = parser.parse_non_stream(content)
|
||||
for call_info in call_info_list:
|
||||
item = self._make_tool_call_item(
|
||||
call_info.name,
|
||||
call_info.parameters or "",
|
||||
custom_names,
|
||||
namespace_names,
|
||||
client_tool_search=self._client_tool_search(request),
|
||||
tool_call_items.append(
|
||||
self._make_tool_call_item(
|
||||
call_info.name,
|
||||
call_info.parameters or "",
|
||||
custom_names,
|
||||
namespace_names,
|
||||
)
|
||||
)
|
||||
self._record_called_name(request, item.call_id, call_info.name)
|
||||
tool_call_items.append(item)
|
||||
parsed_via_native = bool(call_info_list)
|
||||
except Exception as e:
|
||||
logger.error("Tool call parsing error: %s", e)
|
||||
@@ -1412,10 +1268,6 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
def _tool_call_parsing_enabled(
|
||||
self, request: ResponsesRequest, chat_tools: list[Tool]
|
||||
) -> bool:
|
||||
if request._tool_state.chat_tools is not None and request.tool_choice == "none":
|
||||
# The history showed the model tools, so it can still write a call;
|
||||
# parsing drops it, where skipping the parser would return its markup.
|
||||
return bool(self.tool_call_parser)
|
||||
return bool(
|
||||
chat_tools and self.tool_call_parser and request.tool_choice != "none"
|
||||
)
|
||||
@@ -1474,213 +1326,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
return tool_choice
|
||||
return {"type": "function", "function": {"name": tool_choice["name"]}}
|
||||
|
||||
def _renders_natively(self, request: ResponsesRequest) -> bool:
|
||||
"""Whether the served encoding renders the request in its own wire form,
|
||||
which tool search, namespace tools and deferred tools take."""
|
||||
return self.native_tool_loading and (
|
||||
any(
|
||||
tool.type == "namespace" or _defers_loading(tool)
|
||||
for tool in request.tools
|
||||
)
|
||||
or self._uses_tool_search(request)
|
||||
)
|
||||
|
||||
def _uses_tool_search(self, request: ResponsesRequest) -> bool:
|
||||
"""Whether the request searches for tools, or its history has."""
|
||||
if any(tool.type == TOOL_SEARCH for tool in request.tools):
|
||||
return True
|
||||
return any(
|
||||
isinstance(item, dict) and item.get("type") in TOOL_SEARCH_ITEMS
|
||||
for item in self._history_items(request)
|
||||
)
|
||||
|
||||
def _history_items(self, request: ResponsesRequest) -> list:
|
||||
"""The input history, before `previous_response_id` is known to exist."""
|
||||
stored = self.msg_store.get(request.previous_response_id) or []
|
||||
return list(stored) + (request.input if isinstance(request.input, list) else [])
|
||||
|
||||
def _native_tools_error(self, request: ResponsesRequest) -> Optional[str]:
|
||||
deferred_types = {
|
||||
tool.name: tool.type
|
||||
for tool in request.tools
|
||||
if tool.type in ("function", "custom") and tool.defer_loading
|
||||
}
|
||||
for tool in request.tools:
|
||||
if (
|
||||
tool.type == TOOL_SEARCH
|
||||
and (tool.model_extra or {}).get("execution") != "client"
|
||||
):
|
||||
return (
|
||||
'tool_search requires execution "client": this server runs no '
|
||||
"searches"
|
||||
)
|
||||
if tool.type == "namespace" and not _holds_function_tools(tool.tools):
|
||||
return NAMESPACE_MEMBERS_ERROR
|
||||
for item in self._history_items(request):
|
||||
if not isinstance(item, dict) or item.get("type") != "tool_search_output":
|
||||
continue
|
||||
tools = item.get("tools")
|
||||
if not isinstance(tools, list) or not all(
|
||||
isinstance(tool, dict)
|
||||
and tool.get("type") in ("function", "namespace", "custom")
|
||||
for tool in tools
|
||||
):
|
||||
return (
|
||||
"tool_search_output tools must be a list of function, namespace "
|
||||
"or custom tools"
|
||||
)
|
||||
for tool in tools:
|
||||
if tool["type"] == "namespace" and not _holds_function_tools(
|
||||
tool.get("tools")
|
||||
):
|
||||
return NAMESPACE_MEMBERS_ERROR
|
||||
# The served encoding renders no other custom tool.
|
||||
if tool["type"] == "custom" and tool.get("name") != APPLY_PATCH:
|
||||
return (
|
||||
f"tool_search_output can load only the {APPLY_PATCH!r} "
|
||||
"custom tool"
|
||||
)
|
||||
# A deferred declaration is loaded as the tool it declares.
|
||||
declared = deferred_types.get(tool.get("name"), tool["type"])
|
||||
if tool["type"] != "namespace" and declared != tool["type"]:
|
||||
return (
|
||||
f"tool_search_output returns {tool.get('name')!r} as a "
|
||||
f"{tool['type']} tool, but tools declares it as a deferred "
|
||||
f"{declared} tool"
|
||||
)
|
||||
return None
|
||||
|
||||
def _prepare_native_responses_prompt(
|
||||
self, request: ResponsesRequest, chat_request: ChatCompletionRequest
|
||||
) -> None:
|
||||
"""Render the request's own wire form into `chat_request`, whose tools and
|
||||
choice become the ones the history has loaded."""
|
||||
choice = request.effective_tool_choice()
|
||||
tools, items = _defer_tool_loading(
|
||||
[tool.model_dump(exclude_unset=True) for tool in request.tools],
|
||||
[
|
||||
item
|
||||
for item in map(
|
||||
self._native_input_item, self._response_input_history(request)
|
||||
)
|
||||
if item is not None
|
||||
],
|
||||
)
|
||||
body = {"input": items, "tools": tools}
|
||||
if request.instructions:
|
||||
body["instructions"] = request.instructions
|
||||
if choice == "none":
|
||||
body["tool_choice"] = "none"
|
||||
prompt = self.prepare_native_prompt(
|
||||
chat_request, protocol="responses", body=body
|
||||
)
|
||||
tools = prompt.tools
|
||||
request._tool_state = msgspec.structs.replace(
|
||||
request._tool_state,
|
||||
chat_tools=tools,
|
||||
names={
|
||||
call_name: {"namespace": namespace, "name": name}
|
||||
for call_name, (namespace, name) in prompt.namespace_names.items()
|
||||
},
|
||||
discovered_custom_names=frozenset(
|
||||
tool["name"]
|
||||
for item in body["input"]
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
for tool in item["tools"]
|
||||
if tool["type"] == "custom"
|
||||
),
|
||||
)
|
||||
if isinstance(choice, dict):
|
||||
name = choice["name"]
|
||||
if "namespace" in choice:
|
||||
# `tools` keeps conversion order, declared before discovered, so
|
||||
# the last match is the member's latest definition.
|
||||
member = (choice["namespace"], name)
|
||||
name = next(
|
||||
(
|
||||
tool.function.name
|
||||
for tool in reversed(tools)
|
||||
if prompt.namespace_names.get(tool.function.name) == member
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not any(tool.function.name == name for tool in tools) or (
|
||||
choice["type"] == "custom"
|
||||
and name not in self._custom_tool_names(request)
|
||||
):
|
||||
raise ResponseToolChoiceError(
|
||||
f"Tool {choice['name']!r} is not declared in tools without "
|
||||
"defer_loading or loaded by a tool_search_output"
|
||||
)
|
||||
chat_request.tool_choice = ToolChoice(
|
||||
function=ToolChoiceFuncName(name=name)
|
||||
)
|
||||
elif choice == "required" and not tools:
|
||||
raise ResponseToolChoiceError(
|
||||
'tool_choice="required" needs a tool the model can call'
|
||||
)
|
||||
else:
|
||||
chat_request.tool_choice = choice if tools else "none"
|
||||
|
||||
@classmethod
|
||||
def _native_input_item(cls, item: Any) -> Any:
|
||||
if isinstance(item, BaseModel):
|
||||
item = item.model_dump(exclude_none=True)
|
||||
if isinstance(item, dict) and item.get("type") == "reasoning":
|
||||
# recipe reads only `content`; keep the text SGLang replays from a
|
||||
# summary or a store=false client's state.
|
||||
message = cls._normalize_response_message_for_chat(item)
|
||||
if message is None:
|
||||
return None
|
||||
return {
|
||||
"type": "reasoning",
|
||||
"content": [
|
||||
{"type": "reasoning_text", "text": message["reasoning_content"]}
|
||||
],
|
||||
}
|
||||
return item
|
||||
|
||||
@staticmethod
|
||||
def _record_called_name(request: ResponsesRequest, call_id: str, name: str) -> None:
|
||||
"""Keep the name the model called a namespace member by: the item reports
|
||||
its namespace and own name, which several call names can share."""
|
||||
if request._tool_state.chat_tools is not None and (
|
||||
name in request._tool_state.names
|
||||
):
|
||||
request._called_tool_names[call_id] = name
|
||||
|
||||
@staticmethod
|
||||
def _replayed_output_items(request: ResponsesRequest, output: list) -> list[dict]:
|
||||
"""`output` as the history a later request replays: a namespace member's
|
||||
call under the name the model called it by."""
|
||||
items = []
|
||||
for item in output:
|
||||
item = item.model_dump(exclude_none=True)
|
||||
called_name = request._called_tool_names.get(item.get("call_id"))
|
||||
if called_name is not None:
|
||||
item = {**item, "name": called_name}
|
||||
del item["namespace"]
|
||||
items.append(item)
|
||||
return items
|
||||
|
||||
@staticmethod
|
||||
def _custom_tool_names(request: ResponsesRequest) -> set[str]:
|
||||
declared = custom_tool_names(request.tools)
|
||||
if request._tool_state.chat_tools is not None:
|
||||
# A deferred declaration counts once a search result has loaded it.
|
||||
declared -= {tool.name for tool in request.tools if tool.defer_loading}
|
||||
return declared | request._tool_state.discovered_custom_names
|
||||
|
||||
@staticmethod
|
||||
def _client_tool_search(request: ResponsesRequest) -> bool:
|
||||
"""Whether a `tool_search` call is the client-executed search tool."""
|
||||
return request._tool_state.chat_tools is not None and any(
|
||||
tool.type == TOOL_SEARCH for tool in request.tools
|
||||
)
|
||||
|
||||
def _response_tools_to_chat_tools(self, request: ResponsesRequest) -> list[Tool]:
|
||||
if request._tool_state.chat_tools is not None:
|
||||
return list(request._tool_state.chat_tools)
|
||||
# ``function`` and ``custom`` tools flow to chat; built-ins go through
|
||||
# harmony. A custom tool is shimmed into a single-string function tool.
|
||||
chat_tools = []
|
||||
@@ -2487,7 +2133,6 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
tool_follows = opens_item and item_type in (
|
||||
"function_call",
|
||||
"custom_tool_call",
|
||||
"tool_search_call",
|
||||
)
|
||||
terminal = event_type in (
|
||||
"response.completed",
|
||||
@@ -2570,19 +2215,14 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
custom_names = self._custom_tool_names(request)
|
||||
custom_names = custom_tool_names(request.tools)
|
||||
namespace_names = (
|
||||
namespace_function_names(request.tools)
|
||||
if self.tool_call_parser == "muse"
|
||||
else {
|
||||
call_name: (entry.get("namespace"), entry.get("name", call_name))
|
||||
for call_name, entry in request._tool_state.names.items()
|
||||
if entry.get("namespace")
|
||||
}
|
||||
else {}
|
||||
)
|
||||
tool_choice = request.effective_tool_choice()
|
||||
required_json = self._uses_required_tool_json(request, chat_tools)
|
||||
client_tool_search = self._client_tool_search(request)
|
||||
encoding_output = self._encoding_output()
|
||||
token_stream = (
|
||||
encoding_output.new_responses_stream(
|
||||
@@ -2813,13 +2453,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
return []
|
||||
arguments = state["arguments"]
|
||||
events: list = []
|
||||
done_event = openai_responses_types.ResponseOutputItemDoneEvent
|
||||
if state["search"]:
|
||||
completed_item = _tool_search_call(
|
||||
state["item_id"], state["call_id"], arguments
|
||||
)
|
||||
done_event = ResponseToolSearchCallEvent
|
||||
elif state["custom"]:
|
||||
if state["custom"]:
|
||||
payload = decode_custom_tool_input(arguments)
|
||||
# Deltas cannot be retracted, so a payload that no longer
|
||||
# extends what was already streamed defers to the streamed text.
|
||||
@@ -2886,7 +2520,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
events.append(
|
||||
_send_event(
|
||||
done_event(
|
||||
openai_responses_types.ResponseOutputItemDoneEvent(
|
||||
type="response.output_item.done",
|
||||
sequence_number=-1,
|
||||
output_index=state["output_index"],
|
||||
@@ -3083,43 +2717,27 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
current_output_index += 1
|
||||
name = call.name or ""
|
||||
is_custom = name in custom_names
|
||||
is_search = client_tool_search and name == TOOL_SEARCH
|
||||
namespace, name = namespace_names.get(name, (None, name))
|
||||
prefix = (
|
||||
"ctc" if is_custom else "tsc" if is_search else "fc"
|
||||
)
|
||||
state = {
|
||||
"item_id": f"{prefix}_{random_uuid()[:8]}",
|
||||
"item_id": (
|
||||
f"ctc_{random_uuid()[:8]}"
|
||||
if is_custom
|
||||
else f"fc_{random_uuid()[:8]}"
|
||||
),
|
||||
"call_id": f"call_{random_uuid()[:24]}",
|
||||
"output_index": current_output_index,
|
||||
"name": name,
|
||||
"namespace": namespace,
|
||||
"arguments": "",
|
||||
"custom": is_custom,
|
||||
"search": is_search,
|
||||
"payload": "",
|
||||
"added": False,
|
||||
"done": False,
|
||||
}
|
||||
tool_call_states[tool_index] = state
|
||||
self._record_called_name(
|
||||
request, state["call_id"], call.name or ""
|
||||
)
|
||||
if not state["added"]:
|
||||
state["added"] = True
|
||||
added_event = (
|
||||
openai_responses_types.ResponseOutputItemAddedEvent
|
||||
)
|
||||
if state["search"]:
|
||||
# Search arguments are reported whole when it completes.
|
||||
added_item = ResponseToolSearchCall(
|
||||
id=state["item_id"],
|
||||
call_id=state["call_id"],
|
||||
arguments={},
|
||||
status="in_progress",
|
||||
)
|
||||
added_event = ResponseToolSearchCallEvent
|
||||
elif state["custom"]:
|
||||
if state["custom"]:
|
||||
added_item = ResponseCustomToolCall(
|
||||
type="custom_tool_call",
|
||||
id=state["item_id"],
|
||||
@@ -3138,7 +2756,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
**_namespace_field(state),
|
||||
)
|
||||
yield _send_event(
|
||||
added_event(
|
||||
openai_responses_types.ResponseOutputItemAddedEvent(
|
||||
type="response.output_item.added",
|
||||
sequence_number=-1,
|
||||
output_index=state["output_index"],
|
||||
@@ -3168,7 +2786,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
delta=delta,
|
||||
)
|
||||
)
|
||||
elif not state["search"]:
|
||||
else:
|
||||
yield _send_event(
|
||||
openai_responses_types.ResponseFunctionCallArgumentsDeltaEvent(
|
||||
type="response.function_call_arguments.delta",
|
||||
@@ -3318,7 +2936,10 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
self.response_store[final_response.id] = final_response
|
||||
self.msg_store[final_response.id] = self._response_input_history(
|
||||
request
|
||||
) + self._replayed_output_items(request, final_response.output)
|
||||
) + [
|
||||
item.model_dump(exclude_none=True)
|
||||
for item in final_response.output
|
||||
]
|
||||
|
||||
response_dict = _sanitize_response_dict(final_response.model_dump())
|
||||
|
||||
|
||||
@@ -1719,6 +1719,9 @@ class Envs:
|
||||
# Prefill-graph shapes of at least this many tokens run the tail from the tail
|
||||
# graphs in a break (measured crossover 1536-2048 tokens on GB300 TP4).
|
||||
SGLANG_DSV4_DECODER_TAIL_GRAPH_MIN_PREFILL_TOKENS = EnvInt(2048)
|
||||
# Decoder SWA bounded replay: a prefill chunk runs the tail layers only over its
|
||||
# rows in the prompt's last window, else over its last row.
|
||||
SGLANG_OPT_DSV41_PROMPT_WINDOW_TAIL = EnvBool(False)
|
||||
# Paged KV layout of the DeepSeek-V4 family pools: "v4" (584 B/token, every
|
||||
# GPU), "v41" (the SM100 FlashMLA V4.1 formats: 528 B fp8 SWA cache, fp8 or
|
||||
# fp4 compressed caches), "v41_new" (the same bytes interleaved per token, for
|
||||
|
||||
@@ -1024,7 +1024,10 @@ def late_layer_num_tokens(forward_batch: ForwardBatch, *, dspark: bool) -> int:
|
||||
whole_extend = late_layer_whole_extend(forward_batch, dspark=dspark)
|
||||
return sum(
|
||||
late_layer_tail_lens(
|
||||
forward_batch.extend_seq_lens_cpu, SWA_WINDOW, whole_extend
|
||||
forward_batch.extend_seq_lens_cpu,
|
||||
SWA_WINDOW,
|
||||
whole_extend,
|
||||
prompt_rest_lens=forward_batch.extend_prompt_rest_lens_cpu,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1834,6 +1837,7 @@ class DeepseekV4AttnBackend(
|
||||
whole_extend=late_layer_whole_extend(
|
||||
forward_batch, dspark=self.model_runner.spec_algorithm.is_dspark()
|
||||
),
|
||||
prompt_rest_lens=forward_batch.extend_prompt_rest_lens_cpu,
|
||||
)
|
||||
if self.encoder_row_floor is not None:
|
||||
# Folded hits: a tail row never reads below its request's replay start.
|
||||
|
||||
@@ -2252,6 +2252,7 @@ class DeepseekV4HipRadixBackend(
|
||||
tail_len=SWA_WINDOW,
|
||||
device=device,
|
||||
whole_extend=late_layer_whole_extend(forward_batch, dspark=self.is_dspark),
|
||||
prompt_rest_lens=forward_batch.extend_prompt_rest_lens_cpu,
|
||||
)
|
||||
contiguous_start = (
|
||||
extend_lens_cpu[0] - tail_lens_cpu[0] if len(extend_lens_cpu) == 1 else None
|
||||
|
||||
@@ -18,7 +18,7 @@ import logging
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from queue import Empty, Full, Queue
|
||||
from queue import Empty, Queue
|
||||
from typing import TYPE_CHECKING, Any, Callable, List, NamedTuple, Optional
|
||||
|
||||
import torch
|
||||
@@ -42,9 +42,7 @@ from sglang.srt.layers.dp_attention import (
|
||||
)
|
||||
from sglang.srt.mem_cache.l2_transfer import L2Transfer, L2TransferEngine
|
||||
from sglang.srt.mem_cache.memory_pool import MLATokenToKVPool
|
||||
from sglang.srt.mem_cache.pool_host.base import uses_shared_host_layout
|
||||
from sglang.srt.mem_cache.utils import get_storage_hash_str
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.runtime_context import get_parallel
|
||||
from sglang.srt.utils import get_device_module
|
||||
|
||||
@@ -64,31 +62,6 @@ def storage_model_name(
|
||||
return f"{model_name}-{tag}" if model_name else tag
|
||||
|
||||
|
||||
# [HC_PARALLEL_PATCH] knobs for the pipelined HiCache prefetch/backup lanes
|
||||
import os as _hc_os
|
||||
_HC_PREFETCH_WORKERS = int(_hc_os.getenv("SGLANG_HICACHE_PREFETCH_IO_THREADS", "1") or 1)
|
||||
_HC_BACKUP_WORKERS = int(_hc_os.getenv("SGLANG_HICACHE_BACKUP_IO_THREADS", "1") or 1)
|
||||
_HC_TIMING = _hc_os.getenv("SGLANG_HICACHE_TIMING", "0") not in ("", "0", "false", "False")
|
||||
|
||||
# [ASYNC_BACKUP] Per-op pinned index copy + worker-side merge/submit.
|
||||
# Only the direct + page_first_direct shape (the one that does a blocking
|
||||
# device_indices.cpu() on the scheduler thread) is eligible.
|
||||
_ASYNC_BACKUP_SUBMIT = _hc_os.getenv(
|
||||
"SGLANG_ASYNC_BACKUP_SUBMIT", "0"
|
||||
).strip().lower() not in ("", "0", "false", "no", "off")
|
||||
_ASYNC_BACKUP_ENQUEUE_TIMEOUT = float(
|
||||
_hc_os.getenv("SGLANG_ASYNC_BACKUP_ENQUEUE_TIMEOUT", "60") or 60
|
||||
)
|
||||
_ASYNC_BACKUP_IDLE_TIMEOUT = float(
|
||||
_hc_os.getenv("SGLANG_ASYNC_BACKUP_IDLE_TIMEOUT", "120") or 120
|
||||
)
|
||||
_ASYNC_BACKUP_PINNED_BYTES_MAX = int(
|
||||
_hc_os.getenv("SGLANG_ASYNC_BACKUP_PINNED_MB", "2048") or 2048
|
||||
) * 1024 * 1024
|
||||
_ASYNC_BACKUP_MAX_OPS = int(
|
||||
_hc_os.getenv("SGLANG_ASYNC_BACKUP_MAX_OPS", "256") or 256
|
||||
)
|
||||
|
||||
class LayerLoadingEvent:
|
||||
def __init__(self, num_layers: int):
|
||||
self._num_layers = num_layers
|
||||
@@ -152,9 +125,6 @@ class CacheOperation:
|
||||
self.node_ids = [node_id]
|
||||
self.data = None
|
||||
self.pool_transfers = pool_transfers
|
||||
# Set when SGLANG_ASYNC_BACKUP_SUBMIT copied this op's device indices
|
||||
# into pinned host buffers (see _start_async_index_copy).
|
||||
self.async_parts = None
|
||||
|
||||
self.id = CacheOperation.counter
|
||||
CacheOperation.counter += 1
|
||||
@@ -215,39 +185,6 @@ class CacheOperation:
|
||||
return self.priority < other.priority
|
||||
|
||||
|
||||
@dataclass
|
||||
class AsyncIndexCopy:
|
||||
"""One device->pinned-host index copy started where the backup op was made."""
|
||||
|
||||
key: tuple
|
||||
src: torch.Tensor
|
||||
dst: torch.Tensor
|
||||
nbytes: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class AsyncIndexParts:
|
||||
"""Per-op pinned index parts plus the event that guarantees they are final."""
|
||||
|
||||
copies: List[AsyncIndexCopy]
|
||||
ready_event: object
|
||||
nbytes: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class WriteSubmission:
|
||||
ops: List[CacheOperation]
|
||||
tracer: object
|
||||
enqueue_mono_ns: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class WriteResult:
|
||||
ack: Optional[HiCacheAck]
|
||||
ops: List[CacheOperation]
|
||||
error: Optional[BaseException]
|
||||
|
||||
|
||||
class HiCacheAck(NamedTuple):
|
||||
start_event: device_module.Event
|
||||
finish_event: device_module.Event
|
||||
@@ -439,21 +376,6 @@ class HiCacheController:
|
||||
self.write_fence_stream = None
|
||||
self.ack_write_queue: List[HiCacheAck] = []
|
||||
|
||||
# [ASYNC_BACKUP] Pinned index copies are started where the backup op is
|
||||
# created; a single worker waits the per-op events, merges the CPU
|
||||
# index parts and submits the KV copy. Queues are bounded by the state
|
||||
# lock/condition below; write_queue/ack_write_queue stay scheduler-only.
|
||||
self._indices_stream = None
|
||||
self._write_submit_queue: Optional[Queue] = None
|
||||
self._write_result_queue: Queue = Queue()
|
||||
self._write_submit_stop = threading.Event()
|
||||
self._write_submit_thread: Optional[threading.Thread] = None
|
||||
self._write_submit_state_lock = threading.Lock()
|
||||
self._write_submit_cond = threading.Condition(self._write_submit_state_lock)
|
||||
self._write_submit_active = 0
|
||||
self._async_pinned_bytes = 0
|
||||
self._retire_parts: List["AsyncIndexParts"] = []
|
||||
|
||||
self.l2_transfer_engine = L2TransferEngine(io_backend)
|
||||
|
||||
# If a storage backend is provided at startup, treat it as an implicit attach,
|
||||
@@ -779,9 +701,6 @@ class HiCacheController:
|
||||
Requirement: no in-flight requests. This will stop storage threads and release
|
||||
the backend instance (best-effort close).
|
||||
"""
|
||||
# Async backup submissions own pinned index buffers and ACKs; stop them
|
||||
# before any queue/pool teardown so no result is lost.
|
||||
self._stop_write_submission_thread()
|
||||
# Idempotent cleanup: even if `enable_storage` is already False,
|
||||
# we may still have leftover resources (threads/backend/process group) from a
|
||||
# previous partial detach. We attempt cleanup whenever possible.
|
||||
@@ -896,7 +815,6 @@ class HiCacheController:
|
||||
def reset(self):
|
||||
# Reuse detach's queue wakeups and bounded joins, and fail if any
|
||||
# storage thread remains alive before clearing shared state.
|
||||
self._stop_write_submission_thread()
|
||||
self._stop_storage_threads()
|
||||
|
||||
self.write_queue.clear()
|
||||
@@ -936,311 +854,13 @@ class HiCacheController:
|
||||
|
||||
def has_inflight_device_transfers(self) -> bool:
|
||||
"""Whether queued or unacknowledged L2 transfers still use device rows."""
|
||||
with self._write_submit_state_lock:
|
||||
async_pending = (
|
||||
self._write_submit_active > 0
|
||||
or self._async_pinned_bytes > 0
|
||||
or not self._write_result_queue.empty()
|
||||
)
|
||||
return bool(
|
||||
self.write_queue
|
||||
or self.load_queue
|
||||
or self.ack_write_queue
|
||||
or self.ack_load_queue
|
||||
or async_pending
|
||||
)
|
||||
|
||||
# ---- async backup submission -----------------------------------------
|
||||
def async_backup_enabled(self) -> bool:
|
||||
return (
|
||||
_ASYNC_BACKUP_SUBMIT
|
||||
and self.io_backend == "direct"
|
||||
and getattr(self.mem_pool_host, "layout", None) == "page_first_direct"
|
||||
)
|
||||
|
||||
def _async_submit_applicable(self) -> bool:
|
||||
return (
|
||||
self.async_backup_enabled()
|
||||
and len(self.write_queue) <= _ASYNC_BACKUP_MAX_OPS
|
||||
and all(op.async_parts is not None for op in self.write_queue)
|
||||
)
|
||||
|
||||
def _ensure_indices_stream(self):
|
||||
if self._indices_stream is None:
|
||||
self._indices_stream = device_module.Stream()
|
||||
return self._indices_stream
|
||||
|
||||
def _needs_device_index_copy(self, host_pool, indices) -> bool:
|
||||
if indices is None or not getattr(indices, "is_cuda", False):
|
||||
return False
|
||||
# move_indices() returns shared-layout ids untouched; converting them
|
||||
# here would hand the transfer a different interface than the sync path.
|
||||
return not uses_shared_host_layout(host_pool)
|
||||
|
||||
def _async_index_targets(self, op: CacheOperation):
|
||||
targets = []
|
||||
if self._needs_device_index_copy(
|
||||
self.mem_pool_host.anchor_entry.host_pool, op.device_indices
|
||||
):
|
||||
targets.append(((None, None), op.device_indices))
|
||||
for transfer in op.pool_transfers or []:
|
||||
if transfer.device_indices is None:
|
||||
continue
|
||||
entry = self.mem_pool_host.entry_map.get(transfer.name)
|
||||
if entry is None:
|
||||
continue
|
||||
if not self._needs_device_index_copy(
|
||||
entry.host_pool, transfer.device_indices
|
||||
):
|
||||
continue
|
||||
key = (transfer.name, transfer.indices_from_pool)
|
||||
targets.append((key, transfer.device_indices))
|
||||
return targets
|
||||
|
||||
def _start_async_index_copy(self, op: CacheOperation) -> None:
|
||||
"""Start the pinned index D2H copy now, off the scheduler stream.
|
||||
|
||||
Runs in the scheduler thread at backup-op creation time. The copy runs
|
||||
on a dedicated stream ordered after this instant, so it does not inherit
|
||||
later fences (the WAR barrier of a later forward) and the scheduler
|
||||
never blocks in ``.cpu()``.
|
||||
"""
|
||||
if not self.async_backup_enabled():
|
||||
return
|
||||
targets = self._async_index_targets(op)
|
||||
if not targets:
|
||||
return
|
||||
nbytes = sum(int(t.numel()) * int(t.element_size()) for _, t in targets)
|
||||
with self._write_submit_state_lock:
|
||||
if self._async_pinned_bytes + nbytes > _ASYNC_BACKUP_PINNED_BYTES_MAX:
|
||||
return
|
||||
self._async_pinned_bytes += nbytes
|
||||
try:
|
||||
stream = self._ensure_indices_stream()
|
||||
source_stream = device_module.current_stream()
|
||||
copies: List[AsyncIndexCopy] = []
|
||||
with device_module.StreamContext(stream):
|
||||
stream.wait_stream(source_stream)
|
||||
for key, src in targets:
|
||||
# Keep the device rows alive until the copy has run, and pin
|
||||
# the destination so copy_ is a real asynchronous D2H.
|
||||
src.record_stream(stream)
|
||||
dst = torch.empty(
|
||||
src.shape,
|
||||
dtype=src.dtype,
|
||||
device="cpu",
|
||||
pin_memory=True,
|
||||
)
|
||||
dst.copy_(src, non_blocking=True)
|
||||
copies.append(
|
||||
AsyncIndexCopy(
|
||||
key=key,
|
||||
src=src,
|
||||
dst=dst,
|
||||
nbytes=int(src.numel()) * int(src.element_size()),
|
||||
)
|
||||
)
|
||||
ready_event = device_module.Event()
|
||||
ready_event.record(stream)
|
||||
except BaseException:
|
||||
with self._write_submit_state_lock:
|
||||
self._async_pinned_bytes -= nbytes
|
||||
logger.exception("Async backup index copy failed; keeping sync path")
|
||||
return
|
||||
op.async_parts = AsyncIndexParts(
|
||||
copies=copies, ready_event=ready_event, nbytes=nbytes
|
||||
)
|
||||
|
||||
def _ensure_write_submit_thread(self) -> None:
|
||||
if self._write_submit_queue is None:
|
||||
self._write_submit_queue = Queue(maxsize=64)
|
||||
thread = self._write_submit_thread
|
||||
if thread is not None and thread.is_alive():
|
||||
return
|
||||
self._write_submit_stop.clear()
|
||||
thread = threading.Thread(
|
||||
target=self._write_submit_worker,
|
||||
name="hicache-async-backup",
|
||||
daemon=True,
|
||||
)
|
||||
self._write_submit_thread = thread
|
||||
thread.start()
|
||||
|
||||
def _enqueue_async_write(self, ops, tracer) -> bool:
|
||||
"""Enqueue one submission; False means "fall back to the sync path".
|
||||
|
||||
A full queue or a timeout is not fatal: the caller submits the same ops
|
||||
synchronously. Before returning False we wait for the worker to become
|
||||
idle so the fallback cannot race another submission on the shared
|
||||
device_to_host_stream.
|
||||
"""
|
||||
self._ensure_write_submit_thread()
|
||||
item = WriteSubmission(
|
||||
ops=ops, tracer=tracer, enqueue_mono_ns=time.monotonic_ns()
|
||||
)
|
||||
with self._write_submit_cond:
|
||||
self._write_submit_active += 1
|
||||
try:
|
||||
with batch_telemetry.backup_phase("enqueue"):
|
||||
self._write_submit_queue.put(
|
||||
item, timeout=_ASYNC_BACKUP_ENQUEUE_TIMEOUT
|
||||
)
|
||||
except Full:
|
||||
with self._write_submit_cond:
|
||||
self._write_submit_active -= 1
|
||||
self._write_submit_cond.notify_all()
|
||||
logger.warning(
|
||||
"Async backup submit queue full after %.1fs; falling back to "
|
||||
"the synchronous path",
|
||||
_ASYNC_BACKUP_ENQUEUE_TIMEOUT,
|
||||
)
|
||||
self.wait_write_submissions_idle()
|
||||
return False
|
||||
return True
|
||||
|
||||
def _materialize_async_indices(self, ops) -> None:
|
||||
"""Replace device index tensors with the pinned CPU copies (worker)."""
|
||||
for op in ops:
|
||||
parts = op.async_parts
|
||||
if parts is None:
|
||||
continue
|
||||
mapping = {copy.key: copy.dst for copy in parts.copies}
|
||||
anchor = mapping.get((None, None))
|
||||
if anchor is not None:
|
||||
op.device_indices = anchor
|
||||
for transfer in op.pool_transfers or []:
|
||||
dst = mapping.get((transfer.name, transfer.indices_from_pool))
|
||||
if dst is not None:
|
||||
transfer.device_indices = dst
|
||||
|
||||
def _write_submit_worker(self) -> None:
|
||||
while True:
|
||||
item = self._write_submit_queue.get()
|
||||
if item is None:
|
||||
break
|
||||
tracer = item.tracer
|
||||
previous = batch_telemetry.backup_flush_attach(tracer)
|
||||
try:
|
||||
with batch_telemetry.backup_phase("worker"):
|
||||
for op in item.ops:
|
||||
parts = op.async_parts
|
||||
if parts is not None:
|
||||
parts.ready_event.synchronize()
|
||||
self._materialize_async_indices(item.ops)
|
||||
merged = CacheOperation.merge_ops(item.ops)
|
||||
ack = self._submit_write_operation(merged)
|
||||
result = WriteResult(ack=ack, ops=item.ops, error=None)
|
||||
except BaseException as exc: # noqa: BLE001 - fail closed
|
||||
logger.exception("Async backup submission failed")
|
||||
result = WriteResult(ack=None, ops=item.ops, error=exc)
|
||||
finally:
|
||||
batch_telemetry.backup_flush_detach(previous)
|
||||
self._release_async_parts(item.ops)
|
||||
try:
|
||||
batch_telemetry.backup_flush_end(tracer)
|
||||
except Exception:
|
||||
pass
|
||||
# Publish the result before marking the submission done, so an idle
|
||||
# waiter cannot observe active==0 with the result still missing.
|
||||
self._write_result_queue.put(result)
|
||||
with self._write_submit_cond:
|
||||
self._write_submit_active -= 1
|
||||
self._write_submit_cond.notify_all()
|
||||
|
||||
def _release_async_parts(self, ops) -> None:
|
||||
released = 0
|
||||
for op in ops:
|
||||
parts = op.async_parts
|
||||
if parts is None:
|
||||
continue
|
||||
op.async_parts = None
|
||||
parts.copies.clear()
|
||||
released += parts.nbytes
|
||||
if released:
|
||||
with self._write_submit_state_lock:
|
||||
self._async_pinned_bytes -= released
|
||||
|
||||
def _retire_async_parts(self, ops) -> None:
|
||||
"""Release copies that will not be consumed (sync fallback)."""
|
||||
for op in ops:
|
||||
parts = op.async_parts
|
||||
if parts is not None:
|
||||
self._retire_parts.append(parts)
|
||||
op.async_parts = None
|
||||
self._reap_retired_parts()
|
||||
|
||||
def _reap_retired_parts(self) -> None:
|
||||
if not self._retire_parts:
|
||||
return
|
||||
pending = []
|
||||
for parts in self._retire_parts:
|
||||
try:
|
||||
done = parts.ready_event.query()
|
||||
except Exception:
|
||||
done = True
|
||||
if done:
|
||||
parts.copies.clear()
|
||||
with self._write_submit_state_lock:
|
||||
self._async_pinned_bytes -= parts.nbytes
|
||||
else:
|
||||
pending.append(parts)
|
||||
self._retire_parts = pending
|
||||
|
||||
def drain_write_submission_results(self) -> None:
|
||||
"""Move finished async ACKs into ack_write_queue (scheduler thread)."""
|
||||
while True:
|
||||
try:
|
||||
result = self._write_result_queue.get_nowait()
|
||||
except Empty:
|
||||
break
|
||||
if result.error is not None:
|
||||
raise RuntimeError("async backup submission failed") from result.error
|
||||
if result.ack is not None:
|
||||
self.ack_write_queue.append(result.ack)
|
||||
self._reap_retired_parts()
|
||||
|
||||
def wait_write_submissions_idle(
|
||||
self, timeout: float = _ASYNC_BACKUP_IDLE_TIMEOUT
|
||||
) -> None:
|
||||
deadline = time.monotonic() + timeout
|
||||
with self._write_submit_cond:
|
||||
while self._write_submit_active > 0:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise RuntimeError(
|
||||
"Timed out waiting for async backup submissions"
|
||||
)
|
||||
self._write_submit_cond.wait(remaining)
|
||||
|
||||
def _stop_write_submission_thread(self) -> None:
|
||||
if self._write_submit_thread is None:
|
||||
self.drain_write_submission_results()
|
||||
self._reap_retired_parts()
|
||||
if self._retire_parts:
|
||||
raise RuntimeError("async backup index copies still in flight")
|
||||
return
|
||||
try:
|
||||
self.wait_write_submissions_idle()
|
||||
finally:
|
||||
self._write_submit_stop.set()
|
||||
queue = self._write_submit_queue
|
||||
if queue is not None:
|
||||
try:
|
||||
queue.put(None, timeout=5.0)
|
||||
except Exception:
|
||||
logger.exception("Failed to wake the async backup worker")
|
||||
thread = self._write_submit_thread
|
||||
if thread is not None:
|
||||
thread.join(timeout=30.0)
|
||||
if thread.is_alive():
|
||||
raise RuntimeError("async backup submit worker did not stop")
|
||||
self.drain_write_submission_results()
|
||||
if self._retire_parts:
|
||||
raise RuntimeError("async backup index copies still in flight")
|
||||
self._write_submit_thread = None
|
||||
self._write_submit_queue = None
|
||||
|
||||
|
||||
def write(
|
||||
self,
|
||||
device_indices: torch.Tensor,
|
||||
@@ -1256,144 +876,47 @@ class HiCacheController:
|
||||
host_indices = self.mem_pool_host.alloc(len(device_indices))
|
||||
if host_indices is None:
|
||||
return None
|
||||
op = CacheOperation(host_indices, device_indices, node_id, priority)
|
||||
self._start_async_index_copy(op)
|
||||
self.write_queue.append(op)
|
||||
self.write_queue.append(
|
||||
CacheOperation(host_indices, device_indices, node_id, priority)
|
||||
)
|
||||
self.start_writing()
|
||||
return host_indices
|
||||
|
||||
def start_writing(self) -> None:
|
||||
if len(self.write_queue) == 0:
|
||||
# Empty flushes are the common case when write-through fires every
|
||||
# batch: counted so a small `flush_backups` cannot be explained as
|
||||
# "one big flush per loop" without noticing the empty ones.
|
||||
batch_telemetry.detail_count("flush_calls")
|
||||
batch_telemetry.detail_count("flush_empty")
|
||||
return
|
||||
|
||||
ops = list(self.write_queue)
|
||||
tracer = batch_telemetry.backup_flush_begin()
|
||||
async_owned = False
|
||||
try:
|
||||
batch_telemetry.detail_count("flush_calls")
|
||||
batch_telemetry.detail_count("flush_nonempty")
|
||||
self._note_write_queue(ops, len(self.ack_write_queue))
|
||||
|
||||
if self._async_submit_applicable():
|
||||
if self._enqueue_async_write(ops, tracer):
|
||||
self.write_queue.clear()
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_submission_mode", "async"
|
||||
)
|
||||
# The worker owns the tracer from here on (it emits once).
|
||||
async_owned = True
|
||||
return
|
||||
# Queue full / enqueue timeout: retire the pinned copies and
|
||||
# submit synchronously on this thread. `_enqueue_async_write`
|
||||
# waited for the worker to go idle first, so the two paths
|
||||
# never submit to device_to_host_stream concurrently.
|
||||
batch_telemetry.flush_note("sglang.flush_fallback_sync", 1)
|
||||
self._retire_async_parts(ops)
|
||||
|
||||
# Any parts that were started for this queue are not consumed by the
|
||||
# sync path. Retire them unconditionally (they are released once
|
||||
# their events complete) so the pinned-byte budget never leaks.
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_async_skip_ops",
|
||||
sum(1 for op in ops if op.async_parts is None),
|
||||
)
|
||||
self._retire_async_parts(ops)
|
||||
batch_telemetry.flush_note("sglang.flush_submission_mode", "sync")
|
||||
with batch_telemetry.backup_phase("merge_ops"):
|
||||
op = CacheOperation.merge_ops(self.write_queue)
|
||||
self.write_queue.clear()
|
||||
ack = self._submit_write_operation(op)
|
||||
with batch_telemetry.backup_phase("ack_enqueue"):
|
||||
self.ack_write_queue.append(ack)
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_ack_queue_after", len(self.ack_write_queue)
|
||||
)
|
||||
finally:
|
||||
if not async_owned:
|
||||
batch_telemetry.backup_flush_end(tracer)
|
||||
|
||||
def _note_write_queue(self, ops: List[CacheOperation], ack_before: int) -> None:
|
||||
"""Queue-level flush dimensions (same for the sync and async paths)."""
|
||||
batch_telemetry.flush_note("sglang.flush_write_queue_ops", len(ops))
|
||||
batch_telemetry.flush_note("sglang.flush_io_backend", self.io_backend)
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_layout",
|
||||
str(getattr(self.mem_pool_host, "layout", None) or ""),
|
||||
)
|
||||
batch_telemetry.flush_note("sglang.flush_ack_queue_before", ack_before)
|
||||
|
||||
def _submit_write_operation(self, op: CacheOperation) -> HiCacheAck:
|
||||
"""Move indices, submit the D2H copy and build the ACK.
|
||||
|
||||
Runs on the scheduler thread (sync path) or on the async submit worker
|
||||
(async path, with CPU-resident indices). It must not touch
|
||||
``write_queue`` or ``ack_write_queue``; the caller owns those.
|
||||
"""
|
||||
tokens = len(op.device_indices)
|
||||
indices = op.device_indices
|
||||
batch_telemetry.flush_note("sglang.flush_tokens", int(tokens))
|
||||
batch_telemetry.flush_note("sglang.flush_nodes_merged", len(op.node_ids))
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_sidecar_pools", len(op.pool_transfers or [])
|
||||
)
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_indices_device", str(indices.device.type)
|
||||
)
|
||||
batch_telemetry.flush_note("sglang.flush_indices_dtype", str(indices.dtype))
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_indices_numel", int(indices.numel())
|
||||
)
|
||||
|
||||
with batch_telemetry.backup_phase("move_write"):
|
||||
host_indices, device_indices, pool_transfers = (
|
||||
self._move_write_operation(op)
|
||||
)
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_host_indices_on_cpu",
|
||||
bool(getattr(host_indices, "is_cuda", False) is False),
|
||||
)
|
||||
batch_telemetry.flush_note(
|
||||
"sglang.flush_device_indices_on_cpu",
|
||||
bool(getattr(device_indices, "is_cuda", False) is False),
|
||||
)
|
||||
op = CacheOperation.merge_ops(self.write_queue)
|
||||
host_indices, device_indices, pool_transfers = self._move_write_operation(op)
|
||||
self.write_queue.clear()
|
||||
|
||||
if self.write_fence_stream is not None:
|
||||
# The copy may read KV a forward is still writing: under overlap
|
||||
# scheduling a finished request is cached while the next forward
|
||||
# writes its last token, and an evicted SWA page can belong to an
|
||||
# unfinished checkpoint. Later forwards still overlap with the copy.
|
||||
with batch_telemetry.backup_phase("write_fence"):
|
||||
self.l2_transfer_engine.device_to_host_stream.wait_stream(
|
||||
self.write_fence_stream
|
||||
)
|
||||
|
||||
with batch_telemetry.backup_phase("build_l2"):
|
||||
transfers = self._l2_transfers(
|
||||
host_indices, device_indices, pool_transfers
|
||||
)
|
||||
batch_telemetry.flush_note("sglang.flush_l2_transfers", len(transfers))
|
||||
|
||||
with batch_telemetry.backup_phase("submit_d2h"):
|
||||
completion = self.l2_transfer_engine.submit_device_to_host(transfers)
|
||||
|
||||
with batch_telemetry.backup_phase("set_done_event"):
|
||||
self.mem_pool_device_allocator.set_hicache_transfer_done_event(
|
||||
(id(self), "write"), completion.finish_event
|
||||
self.l2_transfer_engine.device_to_host_stream.wait_stream(
|
||||
self.write_fence_stream
|
||||
)
|
||||
|
||||
return HiCacheAck(
|
||||
start_event=completion.start_event,
|
||||
finish_event=completion.finish_event,
|
||||
node_ids=op.node_ids,
|
||||
num_tokens=len(op.device_indices),
|
||||
timing_enabled=completion.timing_enabled,
|
||||
num_tokens_by_pool=self._num_tokens_by_pool(op),
|
||||
num_bytes=self._transfer_num_bytes(op),
|
||||
completion = self.l2_transfer_engine.submit_device_to_host(
|
||||
self._l2_transfers(host_indices, device_indices, pool_transfers)
|
||||
)
|
||||
|
||||
self.mem_pool_device_allocator.set_hicache_transfer_done_event(
|
||||
(id(self), "write"), completion.finish_event
|
||||
)
|
||||
|
||||
self.ack_write_queue.append(
|
||||
HiCacheAck(
|
||||
start_event=completion.start_event,
|
||||
finish_event=completion.finish_event,
|
||||
node_ids=op.node_ids,
|
||||
num_tokens=len(op.device_indices),
|
||||
timing_enabled=completion.timing_enabled,
|
||||
num_tokens_by_pool=self._num_tokens_by_pool(op),
|
||||
num_bytes=self._transfer_num_bytes(op),
|
||||
)
|
||||
)
|
||||
|
||||
def _transfer_num_bytes(self, op: CacheOperation) -> int:
|
||||
@@ -1425,24 +948,15 @@ class HiCacheController:
|
||||
# move indices to GPU if using kernels, to host if using direct indexing
|
||||
if self.io_backend == "kernel":
|
||||
if not host_indices.is_cuda:
|
||||
with batch_telemetry.backup_phase("indices_h2d"):
|
||||
host_indices = host_indices.to(self.device, non_blocking=True)
|
||||
host_indices = host_indices.to(self.device, non_blocking=True)
|
||||
return host_indices, device_indices
|
||||
elif self.io_backend == "direct":
|
||||
if self.mem_pool_host.layout == "layer_first":
|
||||
with batch_telemetry.backup_phase("indices_cpu"):
|
||||
device_indices = device_indices.cpu()
|
||||
with batch_telemetry.backup_phase("index_sort"):
|
||||
host_indices, idx = host_indices.sort()
|
||||
with batch_telemetry.backup_phase("index_select"):
|
||||
return host_indices, device_indices.index_select(0, idx)
|
||||
device_indices = device_indices.cpu()
|
||||
host_indices, idx = host_indices.sort()
|
||||
return host_indices, device_indices.index_select(0, idx)
|
||||
elif self.mem_pool_host.layout == "page_first_direct":
|
||||
# Synchronous D2H of the token indices: the host wall time of
|
||||
# this call is the scheduler blocking on the copy, not a
|
||||
# device bandwidth measurement.
|
||||
with batch_telemetry.backup_phase("indices_cpu"):
|
||||
device_indices = device_indices.cpu()
|
||||
return host_indices, device_indices
|
||||
return host_indices, device_indices.cpu()
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Unsupported layout {self.mem_pool_host.layout!r} for io backend 'direct'"
|
||||
@@ -1638,11 +1152,7 @@ class HiCacheController:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
def _page_transfer(
|
||||
self,
|
||||
operation: PrefetchOperation,
|
||||
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
|
||||
) -> int:
|
||||
def _page_transfer(self, operation: PrefetchOperation) -> int:
|
||||
# Transfer batch by batch
|
||||
prefix_keys = operation.prefix_keys
|
||||
kv_derived_transfers = [
|
||||
@@ -1692,28 +1202,9 @@ class HiCacheController:
|
||||
completed_tokens=completed_pages * self.page_size,
|
||||
operation=operation,
|
||||
)
|
||||
if ack_sink is None:
|
||||
# Keep the direct-call behavior for existing users. The pipelined
|
||||
# coordinator always supplies a sink so workers never share the queue.
|
||||
self.prefetch_sync_queue.put(ack)
|
||||
else:
|
||||
ack_sink(ack)
|
||||
self.prefetch_sync_queue.put(ack)
|
||||
return completed_pages
|
||||
|
||||
def _transfer_prefetch_and_collect_acks(
|
||||
self, operation: PrefetchOperation
|
||||
) -> List[PrefetchAck]:
|
||||
"""Run one worker without publishing ACKs to the shared sync queue."""
|
||||
acks = []
|
||||
try:
|
||||
self._page_transfer(operation, ack_sink=acks.append)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"HiCache prefetch transfer failed for request %s",
|
||||
operation.request_id,
|
||||
)
|
||||
return acks
|
||||
|
||||
def _page_transfer_kv_batch(
|
||||
self,
|
||||
operation: PrefetchOperation,
|
||||
@@ -1756,86 +1247,24 @@ class HiCacheController:
|
||||
def prefetch_io_aux_func(self):
|
||||
"""
|
||||
Auxiliary function conducting IO operations for prefetching.
|
||||
|
||||
# [HC_PARALLEL_PATCH] pipelined: up to SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
# transfers in flight. Every ACK of one operation is published before any
|
||||
# ACK of the next operation, keeping per-rank collective pairing unchanged.
|
||||
"""
|
||||
import time as _time
|
||||
from collections import deque as _deque
|
||||
from concurrent.futures import ThreadPoolExecutor as _TPE
|
||||
|
||||
K = max(1, _HC_PREFETCH_WORKERS)
|
||||
timing = _HC_TIMING
|
||||
ex = _TPE(max_workers=K, thread_name_prefix="hc-prefetch-io")
|
||||
inflight = _deque()
|
||||
lat = []
|
||||
pages = 0
|
||||
try:
|
||||
while not self.storage_stop_event.is_set():
|
||||
while len(inflight) < K:
|
||||
try:
|
||||
operation = self.prefetch_buffer.get_nowait()
|
||||
except Empty:
|
||||
break
|
||||
if operation is None:
|
||||
continue
|
||||
inflight.append(
|
||||
(
|
||||
operation,
|
||||
_time.perf_counter(),
|
||||
ex.submit(
|
||||
self._transfer_prefetch_and_collect_acks, operation
|
||||
),
|
||||
)
|
||||
while not self.storage_stop_event.is_set():
|
||||
try:
|
||||
operation = self.prefetch_buffer.get(block=True, timeout=1)
|
||||
except Empty:
|
||||
continue
|
||||
if operation is None:
|
||||
continue
|
||||
try:
|
||||
self._page_transfer(operation)
|
||||
finally:
|
||||
self.prefetch_sync_queue.put(
|
||||
PrefetchAck(
|
||||
rid=operation.request_id,
|
||||
completed_req=True,
|
||||
operation=operation,
|
||||
)
|
||||
if not inflight:
|
||||
try:
|
||||
operation = self.prefetch_buffer.get(block=True, timeout=1)
|
||||
except Empty:
|
||||
continue
|
||||
if operation is None:
|
||||
continue
|
||||
inflight.append(
|
||||
(
|
||||
operation,
|
||||
_time.perf_counter(),
|
||||
ex.submit(
|
||||
self._transfer_prefetch_and_collect_acks, operation
|
||||
),
|
||||
)
|
||||
)
|
||||
operation, t0, fut = inflight[0]
|
||||
try:
|
||||
acks = fut.result()
|
||||
except Exception:
|
||||
logger.exception("HiCache prefetch transfer failed")
|
||||
acks = []
|
||||
finally:
|
||||
inflight.popleft()
|
||||
if timing:
|
||||
lat.append((_time.perf_counter() - t0) * 1000.0)
|
||||
pages += len(getattr(operation, "hash_value", []) or [])
|
||||
for ack in acks:
|
||||
self.prefetch_sync_queue.put(ack)
|
||||
self.prefetch_sync_queue.put(
|
||||
PrefetchAck(
|
||||
rid=operation.request_id,
|
||||
completed_req=True,
|
||||
operation=operation,
|
||||
)
|
||||
)
|
||||
if timing and len(lat) >= 100:
|
||||
lat.sort()
|
||||
print("[HC_TIMING] prefetch queue=%d inflight=%d n=%d transfer_ms p50=%.1f p95=%.1f pages=%d"
|
||||
% (self.prefetch_buffer.qsize(), len(inflight), len(lat),
|
||||
lat[len(lat) // 2], lat[int(len(lat) * 0.95)], pages), flush=True)
|
||||
lat = []
|
||||
pages = 0
|
||||
finally:
|
||||
# Detach closes the backend after joining this outer thread. Do not
|
||||
# let pool workers outlive that join and touch released state.
|
||||
ex.shutdown(wait=True)
|
||||
)
|
||||
|
||||
def prefetch_rate_limited(self) -> bool:
|
||||
"""
|
||||
@@ -2014,82 +1443,22 @@ class HiCacheController:
|
||||
"HiCache backup failed after %d tokens", operation.completed_tokens
|
||||
)
|
||||
|
||||
def _should_skip_backup(self) -> bool:
|
||||
"""Whether this rank can ACK without doing any storage IO."""
|
||||
return self.backup_skip
|
||||
|
||||
def backup_thread_func(self):
|
||||
"""
|
||||
Manage backup operations from host memory to storage backend.
|
||||
|
||||
# [HC_PARALLEL_PATCH] pipelined: up to SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
# writes in flight, acked in FIFO order.
|
||||
"""
|
||||
import time as _time
|
||||
from collections import deque as _deque
|
||||
from concurrent.futures import ThreadPoolExecutor as _TPE
|
||||
while not self.storage_stop_event.is_set():
|
||||
try:
|
||||
operation = self.backup_queue.get(block=True, timeout=1)
|
||||
if operation is None:
|
||||
continue
|
||||
|
||||
K = max(1, _HC_BACKUP_WORKERS)
|
||||
timing = _HC_TIMING
|
||||
ex = _TPE(max_workers=K, thread_name_prefix="hc-backup-io")
|
||||
inflight = _deque()
|
||||
lat = []
|
||||
pages = 0
|
||||
|
||||
def submit(operation) -> bool:
|
||||
if self._should_skip_backup():
|
||||
if not self.backup_skip:
|
||||
self._page_backup_or_log(operation)
|
||||
self.ack_backup_queue.put(operation)
|
||||
return False
|
||||
inflight.append(
|
||||
(
|
||||
operation,
|
||||
_time.perf_counter(),
|
||||
ex.submit(self._page_backup_or_log, operation),
|
||||
)
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
while not self.storage_stop_event.is_set():
|
||||
while len(inflight) < K:
|
||||
try:
|
||||
operation = self.backup_queue.get_nowait()
|
||||
except Empty:
|
||||
break
|
||||
if operation is None:
|
||||
continue
|
||||
submit(operation)
|
||||
if not inflight:
|
||||
try:
|
||||
operation = self.backup_queue.get(block=True, timeout=1)
|
||||
except Empty:
|
||||
continue
|
||||
if operation is None:
|
||||
continue
|
||||
if not submit(operation):
|
||||
continue
|
||||
operation, t0, fut = inflight[0]
|
||||
try:
|
||||
fut.result()
|
||||
except Exception:
|
||||
logger.exception("HiCache backup failed")
|
||||
finally:
|
||||
inflight.popleft()
|
||||
if timing:
|
||||
lat.append((_time.perf_counter() - t0) * 1000.0)
|
||||
pages += len(getattr(operation, "hash_value", []) or [])
|
||||
self.ack_backup_queue.put(operation)
|
||||
if timing and len(lat) >= 100:
|
||||
lat.sort()
|
||||
print("[HC_TIMING] backup queue=%d inflight=%d n=%d write_ms p50=%.1f p95=%.1f pages=%d"
|
||||
% (self.backup_queue.qsize(), len(inflight), len(lat),
|
||||
lat[len(lat) // 2], lat[int(len(lat) * 0.95)], pages), flush=True)
|
||||
lat = []
|
||||
pages = 0
|
||||
finally:
|
||||
# The outer thread is joined before backend/process-group teardown.
|
||||
# Wait for every submitted write before reporting that thread stopped.
|
||||
ex.shutdown(wait=True)
|
||||
except Empty:
|
||||
continue
|
||||
|
||||
def prefetch_sync_thread_func(self):
|
||||
"""Synchronize prefetch results across all PP and TP ranks."""
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -201,8 +201,6 @@ class LoadSnapshot(msgspec.Struct, omit_defaults=True):
|
||||
num_running_reqs: int = 0
|
||||
num_waiting_reqs: int = 0
|
||||
num_waiting_uncached_tokens: int = 0
|
||||
queued_total_input_tokens: int = 0
|
||||
drain_rate_tokens_per_s: float = 0.0
|
||||
num_used_tokens: int = 0
|
||||
num_total_tokens: int = 0
|
||||
# num_total_tokens minus tokens still awaiting a KV transfer (equal to it
|
||||
|
||||
@@ -120,7 +120,6 @@ from sglang.srt.mem_cache.common import (
|
||||
)
|
||||
from sglang.srt.mem_cache.memory_pool import HybridReqToTokenPool, ReqToTokenPool
|
||||
from sglang.srt.mem_cache.unified_cache.component_type import ComponentType
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.model_executor.encoder_swa_replay import resets_swa_window
|
||||
from sglang.srt.model_executor.forward_batch_info import (
|
||||
CaptureHiddenMode,
|
||||
@@ -1149,10 +1148,6 @@ class Req(ReqDllmMixin):
|
||||
# Full-KV-derived boundary whose SWA window should be inserted after
|
||||
# the current prefill pass.
|
||||
self.swa_branching_seqlen: Optional[int] = None
|
||||
# Observability: Full-KV tokens rejected by the SWA prefix-match gate.
|
||||
self.swa_gate_reject_tokens: int = 0
|
||||
# Longest Full-KV match before component gates (set by match_kv_cache).
|
||||
self.full_kv_hit_length: int = 0
|
||||
|
||||
# for cross-encoder model
|
||||
self.token_type_ids = token_type_ids
|
||||
@@ -2088,8 +2083,6 @@ class Req(ReqDllmMixin):
|
||||
self.num_matched_prefix_tokens = 0
|
||||
self.lock = None
|
||||
self.swa_branching_seqlen = None
|
||||
self.swa_gate_reject_tokens = 0
|
||||
self.full_kv_hit_length = 0
|
||||
self.extend_end = None
|
||||
self.dllm_initialized = False
|
||||
self.is_retracted = True
|
||||
@@ -2656,6 +2649,8 @@ class ScheduleBatch(ScheduleBatchDisaggregationDecodeMixin):
|
||||
# For DP attention
|
||||
global_num_tokens: Optional[List[int]] = None
|
||||
global_num_tokens_for_logprob: Optional[List[int]] = None
|
||||
# Per DP rank, the rows the decoder SWA tail drops; None when no rank trims.
|
||||
global_decoder_trim_rows: Optional[List[int]] = None
|
||||
# The draft model can use a different MoE A2A backend than the target.
|
||||
draft_global_num_tokens: Optional[List[int]] = None
|
||||
draft_global_num_tokens_for_logprob: Optional[List[int]] = None
|
||||
@@ -2916,12 +2911,9 @@ class ScheduleBatch(ScheduleBatchDisaggregationDecodeMixin):
|
||||
)
|
||||
self.encoder_swa_reset = [resets_swa_window(r) for r in reqs]
|
||||
# Allocate memory
|
||||
with batch_telemetry.detail_phase("alloc_for_extend"):
|
||||
(
|
||||
out_cache_loc,
|
||||
req_pool_indices_tensor,
|
||||
req_pool_indices_cpu,
|
||||
) = alloc_for_extend(self)
|
||||
out_cache_loc, req_pool_indices_tensor, req_pool_indices_cpu = alloc_for_extend(
|
||||
self
|
||||
)
|
||||
|
||||
# Set fields
|
||||
input_embeds = []
|
||||
@@ -3137,11 +3129,10 @@ class ScheduleBatch(ScheduleBatchDisaggregationDecodeMixin):
|
||||
self.prepare_encoder_info_extend(input_ids, seq_lens)
|
||||
|
||||
# Build sampling info
|
||||
with batch_telemetry.detail_phase("sampling_batch_info"):
|
||||
self.sampling_info = SamplingBatchInfo.from_schedule_batch(
|
||||
self,
|
||||
self.model_config.vocab_size,
|
||||
)
|
||||
self.sampling_info = SamplingBatchInfo.from_schedule_batch(
|
||||
self,
|
||||
self.model_config.vocab_size,
|
||||
)
|
||||
|
||||
def _mamba_radix_cache_v2_req_prepare_for_extend(
|
||||
self,
|
||||
|
||||
@@ -28,25 +28,6 @@ logger = logging.getLogger(__name__)
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ==============================================================================
|
||||
|
||||
|
||||
def observe_swa_gate_reject_on_admission(req, admitted_prefix_len: int) -> int:
|
||||
"""Emit the SWA prefix-match gate's cost once per prefill admission.
|
||||
|
||||
``match_kv_cache`` also runs on waiting-queue probes, PD decode and HiCache
|
||||
rematches; only the prefill admission knows the final accepted prefix (after
|
||||
any host load-back), so the counter is incremented here. A retracted and
|
||||
re-admitted request is counted again because it is re-prefilled.
|
||||
"""
|
||||
gap = max(0, getattr(req, "full_kv_hit_length", 0) - admitted_prefix_len)
|
||||
req.swa_gate_reject_tokens = gap
|
||||
if gap > 0:
|
||||
time_stats = getattr(req, "time_stats", None)
|
||||
collector = getattr(time_stats, "metrics_collector", None)
|
||||
observe = getattr(collector, "observe_swa_gate_reject_admission", None)
|
||||
if observe is not None and getattr(time_stats, "enable_metrics", False):
|
||||
observe(gap)
|
||||
return gap
|
||||
"""Request scheduler policy"""
|
||||
|
||||
import os
|
||||
@@ -1641,7 +1622,6 @@ class PrefillAdder:
|
||||
) -> None:
|
||||
assert req.prefix_len == admission.prefix_len
|
||||
req.extend_end = admission.prefix_len + admission.extend_len
|
||||
observe_swa_gate_reject_on_admission(req, admission.prefix_len)
|
||||
req.lock = self.tree_cache.lock(req.last_node)
|
||||
self.can_run_list.append(req)
|
||||
if admission.is_chunked:
|
||||
|
||||
@@ -304,7 +304,6 @@ from sglang.srt.model_executor.forward_batch_info import PPProxyTensors
|
||||
from sglang.srt.model_executor.runner_utils.pool import prewarm_graph_pool_borrow
|
||||
from sglang.srt.model_loader.utils import get_resolved_model_impl
|
||||
from sglang.srt.multiplex.multiplexing_mixin import SchedulerMultiplexMixin
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.metrics_collector import SchedulerMetricsCollector
|
||||
from sglang.srt.observability.req_time_stats import (
|
||||
flush_trace_batch,
|
||||
@@ -645,14 +644,6 @@ class Scheduler(
|
||||
)
|
||||
else:
|
||||
self.decode_offload_manager = None
|
||||
# Prefill read-done lets the schedule stream run ahead of the forward that
|
||||
# writes a stashed chunk. The cache controller's fence streams above order
|
||||
# its copies behind that forward; the decode offload controller has none.
|
||||
self.tp_worker.model_runner.kv_offload_fenced_on_forward_stream = (
|
||||
self.enable_hierarchical_cache
|
||||
and self.tree_cache.cache_controller is not None
|
||||
and self.decode_offload_manager is None
|
||||
)
|
||||
|
||||
# Init running status
|
||||
self.init_running_status()
|
||||
@@ -1425,8 +1416,6 @@ class Scheduler(
|
||||
metrics_collector_context=self.metrics_collector_context,
|
||||
metrics_collector=self.metrics_collector,
|
||||
)
|
||||
# Read the already-running DeviceTimer; adds no CUDA events.
|
||||
batch_telemetry.install_forward_reporter(self)
|
||||
|
||||
def init_schedule_policy(self):
|
||||
# Init schedule policy and new token estimation
|
||||
@@ -2189,10 +2178,6 @@ class Scheduler(
|
||||
# Count successful receive/broadcast time as active; empty polls stay idle.
|
||||
self.metrics_reporter.record_scheduler_active(recv_start_ns)
|
||||
self.process_input_requests(recv_reqs)
|
||||
if recv_reqs:
|
||||
# The last snapshot may predate these requests (it is published after
|
||||
# each forward, before the requests sent during it are read).
|
||||
self.publish_load_snapshot(force=True)
|
||||
return recv_reqs
|
||||
|
||||
@scheduler_stage_method(SCHEDULER_STAGE_PROCESS_REQUESTS)
|
||||
@@ -2578,7 +2563,6 @@ class Scheduler(
|
||||
),
|
||||
get_stats=lambda: self.metrics_reporter.stats,
|
||||
get_chunked_req=lambda: self.chunked_req,
|
||||
get_grammar_queue=lambda: self.grammar_manager.grammar_queue,
|
||||
get_disagg_prefill_bootstrap_queue=lambda: (
|
||||
self.disagg_prefill_bootstrap_queue
|
||||
),
|
||||
@@ -3872,17 +3856,8 @@ class Scheduler(
|
||||
def get_next_batch_to_run(
|
||||
self, running_batch: ScheduleBatch, last_batch: Optional[ScheduleBatch]
|
||||
) -> NextBatchPlan:
|
||||
bt_pass = batch_telemetry.begin_pass()
|
||||
if bt_pass is not None:
|
||||
# Capture before the chunked request can be consumed by this pass.
|
||||
batch_telemetry.note_had_chunked(self.chunked_req is not None)
|
||||
|
||||
self.process_pending_chunked_abort()
|
||||
if bt_pass is not None:
|
||||
_bt_t = time.monotonic()
|
||||
self._process_hicache_events()
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_hicache(time.monotonic() - _bt_t)
|
||||
|
||||
if self.dllm_config is not None:
|
||||
self.dllm_manager.filter_finished_reqs()
|
||||
@@ -3973,11 +3948,7 @@ class Scheduler(
|
||||
# Before merging the new batch into running batch:
|
||||
# 1. All new batches are none -> need_mlp_sync remains true (sync is needed for decode batch).
|
||||
# 2. All new batches are some (prefill / idle) -> we do not need prepare mlp sync one more time.
|
||||
if bt_pass is not None:
|
||||
_bt_t = time.monotonic()
|
||||
new_batch = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(new_batch)
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
|
||||
need_mlp_sync = new_batch is None
|
||||
|
||||
if new_batch is not None:
|
||||
@@ -3996,13 +3967,9 @@ class Scheduler(
|
||||
ret = None
|
||||
|
||||
# Handle DP attention and log stats
|
||||
if bt_pass is not None:
|
||||
_bt_t = time.monotonic()
|
||||
ret = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(
|
||||
ret, need_sync=need_mlp_sync
|
||||
)
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
|
||||
# Decode->extend conversion keeps a heterogeneous dp step replayable.
|
||||
# A converted running batch stays the running batch; next step it comes
|
||||
# back as last_batch and is filtered in place, not merged with itself.
|
||||
@@ -4017,9 +3984,6 @@ class Scheduler(
|
||||
if ret:
|
||||
set_schedule_time_batch(ret)
|
||||
|
||||
if bt_pass is not None:
|
||||
batch_telemetry.end_pass(dp_rank=get_parallel().attn_dp_rank)
|
||||
|
||||
return NextBatchPlan(batch_to_run=ret, running_batch=running_batch)
|
||||
|
||||
def get_num_allocatable_reqs(
|
||||
@@ -4124,18 +4088,11 @@ class Scheduler(
|
||||
return None, running_batch
|
||||
|
||||
# Get priority queue
|
||||
_pf_t0 = time.monotonic_ns()
|
||||
self.policy.calc_priority(
|
||||
self.waiting_queue,
|
||||
running_batch,
|
||||
processed_tokens=self.processed_tokens_counter,
|
||||
)
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline
|
||||
|
||||
fwd_timeline.detail("prefill_policy", _pf_t0, time.monotonic_ns())
|
||||
except Exception:
|
||||
fwd_timeline = None
|
||||
|
||||
if TEST_RETRACT and running_bs > TEST_RETRACT_NO_PREFILL_BS:
|
||||
# If we are testing retraction and the running batch size exceeds
|
||||
@@ -4159,7 +4116,6 @@ class Scheduler(
|
||||
else:
|
||||
prefill_tile_block_m = 64 # Fallback for non-Triton backends
|
||||
|
||||
_pf_t1 = time.monotonic_ns()
|
||||
adder = self.get_prefill_adder_class()(
|
||||
self.page_size,
|
||||
self.tree_cache,
|
||||
@@ -4202,12 +4158,9 @@ class Scheduler(
|
||||
running_batch.reqs,
|
||||
)
|
||||
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail("prefill_adder_init", _pf_t1, time.monotonic_ns())
|
||||
mamba_allocator = getattr(self.req_to_token_pool, "mamba_allocator", None)
|
||||
if mamba_allocator is not None:
|
||||
mamba_allocator.alloc_group_begin(len(self.waiting_queue))
|
||||
_pf_t2 = time.monotonic_ns()
|
||||
buffer_pipeline = self.tree_cache.buffer_pipeline
|
||||
# Get requests from the waiting queue to a new prefill batch
|
||||
for req in self.waiting_queue:
|
||||
@@ -4215,7 +4168,6 @@ class Scheduler(
|
||||
# candidate: the resumed chunk or mixed decode rows can spend it.
|
||||
if adder.budget_state() == AddReqResult.OTHER:
|
||||
break
|
||||
batch_telemetry.mark_candidate()
|
||||
if self.enable_lora and not self.can_schedule_lora_req(req, running_loras):
|
||||
continue
|
||||
|
||||
@@ -4254,7 +4206,6 @@ class Scheduler(
|
||||
req.time_stats.observe_storage_prefetch_ready(prefetch_done)
|
||||
if not prefetch_done:
|
||||
# skip staging requests that are ongoing prefetch
|
||||
batch_telemetry.mark_skipped_prefetch()
|
||||
continue
|
||||
# Pop the L3-loaded span. Unified cache exposes its absolute
|
||||
# start so cache-mode L2/L3 attribution survives L3-tail eviction.
|
||||
@@ -4268,11 +4219,7 @@ class Scheduler(
|
||||
# marks the staged span below once it is surfaced.
|
||||
req.host_hit_is_storage = False
|
||||
|
||||
if batch_telemetry.stage_timing_enabled():
|
||||
_bt_t = time.monotonic()
|
||||
req.init_next_round_input(self.tree_cache)
|
||||
if batch_telemetry.stage_timing_enabled():
|
||||
batch_telemetry.add_match(time.monotonic() - _bt_t)
|
||||
if self.enable_hicache_storage and (
|
||||
self._prefetch_after_device_hit_loss(req)
|
||||
):
|
||||
@@ -4282,15 +4229,11 @@ class Scheduler(
|
||||
req.time_stats.observe_storage_staging_ready(staging_ready)
|
||||
if not staging_ready:
|
||||
continue
|
||||
if batch_telemetry.stage_timing_enabled():
|
||||
_bt_t = time.monotonic()
|
||||
res = adder.add_one_req(
|
||||
req,
|
||||
has_chunked_req=(self.chunked_req is not None),
|
||||
truncation_align_size=self.truncation_align_size,
|
||||
)
|
||||
if batch_telemetry.stage_timing_enabled():
|
||||
batch_telemetry.add_adder(time.monotonic() - _bt_t)
|
||||
|
||||
if self.enable_lora:
|
||||
running_loras.add(req.lora_id)
|
||||
@@ -4320,15 +4263,6 @@ class Scheduler(
|
||||
|
||||
if mamba_allocator is not None:
|
||||
mamba_allocator.alloc_group_end()
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail("prefill_scan", _pf_t2, time.monotonic_ns())
|
||||
_pf_t3 = time.monotonic_ns()
|
||||
_pf_cpu_t3 = (
|
||||
batch_telemetry.thread_cpu_ns()
|
||||
if batch_telemetry.loop_detail_enabled()
|
||||
else None
|
||||
)
|
||||
_had_chunked = self.chunked_req is not None
|
||||
|
||||
# Update waiting queue
|
||||
can_run_list: List[Req] = adder.can_run_list
|
||||
@@ -4337,14 +4271,6 @@ class Scheduler(
|
||||
# though the request they made room for was not admitted.
|
||||
for req in adder.preempt_list:
|
||||
self._add_request_to_queue(req)
|
||||
batch_telemetry.set_admitted(0)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail(
|
||||
"prefill_batch_init",
|
||||
_pf_t3,
|
||||
time.monotonic_ns(),
|
||||
thread_cpu_ns=self._detail_cpu_delta(_pf_cpu_t3),
|
||||
)
|
||||
return None, running_batch
|
||||
|
||||
can_run_set = set(can_run_list)
|
||||
@@ -4368,17 +4294,16 @@ class Scheduler(
|
||||
set_time_batch(can_run_list, "set_forward_entry_time")
|
||||
|
||||
# Create a new batch
|
||||
with batch_telemetry.detail_phase("schedule_batch_init"):
|
||||
new_batch = ScheduleBatch.init_new(
|
||||
can_run_list,
|
||||
self.req_to_token_pool,
|
||||
self.token_to_kv_pool_allocator,
|
||||
self.tree_cache,
|
||||
self.model_config,
|
||||
self.enable_overlap,
|
||||
self.spec_algorithm,
|
||||
chunked_req=self.chunked_req,
|
||||
)
|
||||
new_batch = ScheduleBatch.init_new(
|
||||
can_run_list,
|
||||
self.req_to_token_pool,
|
||||
self.token_to_kv_pool_allocator,
|
||||
self.tree_cache,
|
||||
self.model_config,
|
||||
self.enable_overlap,
|
||||
self.spec_algorithm,
|
||||
chunked_req=self.chunked_req,
|
||||
)
|
||||
|
||||
new_batch.contains_last_prefill_chunk = (
|
||||
self.chunked_req is None or len(can_run_list) != 1
|
||||
@@ -4387,35 +4312,29 @@ class Scheduler(
|
||||
|
||||
if self.enable_hierarchical_cache or self.enable_unified_cache_external_linker:
|
||||
# todo (zhiqiang): disable cuda graph execution if hicache loading triggered
|
||||
with batch_telemetry.detail_phase("ready_to_load_host_cache"):
|
||||
new_batch.hicache_consumer_index = (
|
||||
self.tree_cache.ready_to_load_host_cache()
|
||||
)
|
||||
new_batch.hicache_consumer_index = (
|
||||
self.tree_cache.ready_to_load_host_cache()
|
||||
)
|
||||
|
||||
with batch_telemetry.detail_phase("prepare_for_extend"):
|
||||
new_batch.prepare_for_extend()
|
||||
# Snapshot the prefill-only shape before mixed-chunk folding:
|
||||
# mix_with_running() appends decode rows to extend_num_tokens/prefix_lens,
|
||||
# so reading them after the branch would mix two different scopes.
|
||||
_prefill_n_new = int(getattr(new_batch, "extend_num_tokens", 0) or 0)
|
||||
_prefill_n_cached = int(sum(getattr(new_batch, "prefix_lens", ()) or ()))
|
||||
new_batch.prepare_for_extend()
|
||||
|
||||
if self.tp_worker.model_runner.prefill_aware_swa:
|
||||
for req in can_run_list:
|
||||
req.kv.swa_evict_floor = req.extend_end
|
||||
|
||||
# Record prefill stats for logging after forward.
|
||||
with batch_telemetry.detail_phase("prefill_stats"):
|
||||
new_batch.prefill_stats = PrefillStats.from_adder(
|
||||
adder,
|
||||
running_batch.reqs,
|
||||
self.enable_priority_scheduling,
|
||||
num_pending_tokens=self._prefill_pending_tokens(),
|
||||
)
|
||||
new_batch.prefill_stats = PrefillStats.from_adder(
|
||||
adder,
|
||||
running_batch.reqs,
|
||||
self.enable_priority_scheduling,
|
||||
num_pending_tokens=self.load_inquirer._get_num_pending_tokens(
|
||||
chunk_deduct=(
|
||||
self.chunked_req.extend_len if self.chunked_req is not None else 0
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# Mixed-style chunked prefill
|
||||
_mixed_t0 = None
|
||||
_mixed_cpu0 = None
|
||||
if (
|
||||
self.is_mixed_chunk
|
||||
and not running_batch.is_empty()
|
||||
@@ -4425,9 +4344,6 @@ class Scheduler(
|
||||
# Beam member rows are not supported inside a mixed extend batch.
|
||||
and all(r.beam_group is None for r in running_batch.reqs)
|
||||
):
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
_mixed_t0 = time.monotonic_ns()
|
||||
_mixed_cpu0 = batch_telemetry.thread_cpu_ns()
|
||||
# TODO (lianmin): support return_logprob + mixed chunked prefill
|
||||
finishing_reqs = self._reqs_finishing_inflight(running_batch)
|
||||
running_batch.filter_batch(reqs_to_exclude=finishing_reqs)
|
||||
@@ -4457,71 +4373,8 @@ class Scheduler(
|
||||
else:
|
||||
new_batch.decoding_reqs = None
|
||||
|
||||
batch_telemetry.set_admitted(len(can_run_list))
|
||||
batch_telemetry.note_new_chunk(adder.new_chunked_req is not None)
|
||||
if _mixed_t0 is not None and fwd_timeline is not None:
|
||||
fwd_timeline.detail(
|
||||
"mixed_chunk_tail",
|
||||
_mixed_t0,
|
||||
time.monotonic_ns(),
|
||||
thread_cpu_ns=self._detail_cpu_delta(_mixed_cpu0),
|
||||
)
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
# Prefill-only scope for the three base counters, even when the
|
||||
# batch was folded with running decode rows afterwards.
|
||||
batch_telemetry.detail_attr("prefill_batch_n_reqs", len(can_run_list))
|
||||
batch_telemetry.detail_attr("prefill_batch_n_new", _prefill_n_new)
|
||||
batch_telemetry.detail_attr("prefill_batch_n_cached", _prefill_n_cached)
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_chunked", bool(self.chunked_req is not None)
|
||||
)
|
||||
batch_telemetry.detail_attr("prefill_batch_resumed_chunked", bool(_had_chunked))
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_new_chunked", bool(adder.new_chunked_req is not None)
|
||||
)
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_mixed_chunk", bool(_mixed_t0 is not None)
|
||||
)
|
||||
if _mixed_t0 is not None:
|
||||
# Mixed batches carry a second scope; keep it explicit instead
|
||||
# of folding decode rows into the prefill counters above.
|
||||
_decode_rows = len(getattr(new_batch, "decoding_reqs", ()) or ())
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_decode_reqs", _decode_rows
|
||||
)
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_merged_n_reqs", len(can_run_list) + _decode_rows
|
||||
)
|
||||
batch_telemetry.detail_attr(
|
||||
"prefill_batch_merged_n_new",
|
||||
int(getattr(new_batch, "extend_num_tokens", 0) or 0),
|
||||
)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail(
|
||||
"prefill_batch_init",
|
||||
_pf_t3,
|
||||
time.monotonic_ns(),
|
||||
thread_cpu_ns=self._detail_cpu_delta(_pf_cpu_t3),
|
||||
)
|
||||
return new_batch, running_batch
|
||||
|
||||
@staticmethod
|
||||
def _detail_cpu_delta(cpu_start_ns):
|
||||
"""Thread CPU delta for detail spans; None when the clock is missing."""
|
||||
if cpu_start_ns is None:
|
||||
return None
|
||||
now = batch_telemetry.thread_cpu_ns()
|
||||
return None if now is None else now - cpu_start_ns
|
||||
|
||||
def _prefill_pending_tokens(self) -> int:
|
||||
"""Pending-token evaluation for PrefillStats, timed as its own phase."""
|
||||
with batch_telemetry.detail_phase("pending_tokens"):
|
||||
return self.load_inquirer._get_num_pending_tokens(
|
||||
chunk_deduct=(
|
||||
self.chunked_req.extend_len if self.chunked_req is not None else 0
|
||||
),
|
||||
)
|
||||
|
||||
def can_schedule_lora_req(
|
||||
self, req: Req, running_loras: set[Optional[str]]
|
||||
) -> bool:
|
||||
@@ -4756,9 +4609,8 @@ class Scheduler(
|
||||
|
||||
# PD prefill: early-send cached prefix KV, overlapping the suffix forward.
|
||||
if self.disaggregation_mode == DisaggregationMode.PREFILL:
|
||||
with batch_telemetry.detail_phase("early_send_host"):
|
||||
for req in batch.reqs:
|
||||
self.maybe_send_cached_prefix_chunk(req)
|
||||
for req in batch.reqs:
|
||||
self.maybe_send_cached_prefix_chunk(req)
|
||||
|
||||
# Run forward
|
||||
if self.is_generation:
|
||||
@@ -4785,8 +4637,7 @@ class Scheduler(
|
||||
# mix_running_indices). Run OUTSIDE isolation so the
|
||||
# snapshot captures the post-consume state — restoring
|
||||
# post-forward must not un-consume staging.
|
||||
with batch_telemetry.detail_phase("resolve_inputs"):
|
||||
resolve_forward_inputs(batch, self.future_map)
|
||||
resolve_forward_inputs(batch, self.future_map)
|
||||
|
||||
with self._forward_isolation(batch, overlap=True):
|
||||
future_indices = batch.req_pool_indices
|
||||
@@ -4809,10 +4660,9 @@ class Scheduler(
|
||||
)
|
||||
|
||||
# FIXME: pp is not compatible with overlap
|
||||
with batch_telemetry.detail_phase("model_forward_host"):
|
||||
batch_result = self.model_worker.forward_batch_generation(
|
||||
batch, **fwd_kwargs
|
||||
)
|
||||
batch_result = self.model_worker.forward_batch_generation(
|
||||
batch, **fwd_kwargs
|
||||
)
|
||||
if batch.spec_algorithm.is_none():
|
||||
self.future_map.publish(future_indices, batch.seq_lens + 1)
|
||||
# Park any refs the worker wants kept alive 2 iters
|
||||
@@ -4850,10 +4700,7 @@ class Scheduler(
|
||||
return_hidden_states=batch.return_hidden_states,
|
||||
)
|
||||
else:
|
||||
with batch_telemetry.detail_phase(
|
||||
"copy_result_submit"
|
||||
):
|
||||
self._copy_result_to_cpu(batch_result, batch)
|
||||
self._copy_result_to_cpu(batch_result, batch)
|
||||
else:
|
||||
batch_result.future_indices = future_indices
|
||||
|
||||
@@ -5190,19 +5037,15 @@ class Scheduler(
|
||||
):
|
||||
# Flush async trace ops here: in overlap mode this CPU work runs while
|
||||
# the next batch's GPU forward is in flight, giving free overlap.
|
||||
with batch_telemetry.detail_phase("flush_trace"):
|
||||
flush_trace_batch(batch.reqs)
|
||||
with batch_telemetry.detail_phase("load_snapshot"):
|
||||
snapshot = self.publish_load_snapshot(
|
||||
force=batch.forward_mode.is_extend()
|
||||
)
|
||||
# Router-facing gauge on the dedicated PUB socket, reusing the
|
||||
# snapshot above rather than walking the queues again.
|
||||
self.load_publisher.publish_load_stat(
|
||||
self.load_inquirer.get_loads,
|
||||
force=batch.forward_mode.is_extend(),
|
||||
snapshot=snapshot,
|
||||
)
|
||||
flush_trace_batch(batch.reqs)
|
||||
snapshot = self.publish_load_snapshot(force=batch.forward_mode.is_extend())
|
||||
# Router-facing gauge on the dedicated PUB socket, reusing the
|
||||
# snapshot above rather than walking the queues again.
|
||||
self.load_publisher.publish_load_stat(
|
||||
self.load_inquirer.get_loads,
|
||||
force=batch.forward_mode.is_extend(),
|
||||
snapshot=snapshot,
|
||||
)
|
||||
|
||||
if batch.forward_mode.is_decode():
|
||||
self.batch_result_processor.process_batch_result_decode(batch, result)
|
||||
@@ -5219,28 +5062,19 @@ class Scheduler(
|
||||
self.batch_result_processor.process_batch_result_idle(batch, result)
|
||||
|
||||
# Submit this batch's queued host backups before the next scheduler step.
|
||||
# Tag the origin so the per-flush micro timer can separate these from
|
||||
# the ones issued by check_hicache_events / write().
|
||||
_flush_origin = batch_telemetry.set_flush_origin("process_result")
|
||||
try:
|
||||
with batch_telemetry.detail_phase("flush_backups"):
|
||||
self.tree_cache.flush_pending_backups()
|
||||
finally:
|
||||
batch_telemetry.set_flush_origin(_flush_origin)
|
||||
self.tree_cache.flush_pending_backups()
|
||||
|
||||
with batch_telemetry.detail_phase("step_counters"):
|
||||
self._record_step_counters(batch, result)
|
||||
self._record_step_counters(batch, result)
|
||||
|
||||
with batch_telemetry.detail_phase("metrics_report"):
|
||||
self.metrics_reporter.log_batch_result_stats(batch, result)
|
||||
self.metrics_reporter.log_batch_result_stats(batch, result)
|
||||
|
||||
# Emit forward pass metrics (every iteration when enabled)
|
||||
if self.enable_fpm:
|
||||
self.metrics_reporter._emit_forward_pass_metrics(batch, result)
|
||||
# Emit forward pass metrics (every iteration when enabled)
|
||||
if self.enable_fpm:
|
||||
self.metrics_reporter._emit_forward_pass_metrics(batch, result)
|
||||
|
||||
self._maybe_clear_mm_inputs(batch)
|
||||
self.maybe_send_health_check_signal()
|
||||
self.metrics_reporter.update_device_timer()
|
||||
self._maybe_clear_mm_inputs(batch)
|
||||
self.maybe_send_health_check_signal()
|
||||
self.metrics_reporter.update_device_timer()
|
||||
|
||||
def _record_step_counters(
|
||||
self, batch: ScheduleBatch, result: GenerationBatchResult
|
||||
|
||||
@@ -29,7 +29,6 @@ from sglang.srt.managers.schedule_batch import (
|
||||
ScheduleBatch,
|
||||
mamba_lazy_spec_in_window,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.mem_cache.common import (
|
||||
checkpoint_kv_cache,
|
||||
release_kv_cache,
|
||||
@@ -996,10 +995,6 @@ class SchedulerBatchResultProcessor:
|
||||
if batch.spec_algorithm.is_none() and logits_output is not None:
|
||||
newly_finished_beam_groups = self.beam_coordinator.commit_decode(batch)
|
||||
|
||||
# Keep the hot loop free when telemetry is off: read the gate once and
|
||||
# skip the per-request finished() check entirely.
|
||||
track_completions = batch_telemetry.enabled()
|
||||
finished_reqs = 0
|
||||
for i, req in enumerate(batch.reqs):
|
||||
req: Req
|
||||
|
||||
@@ -1014,8 +1009,6 @@ class SchedulerBatchResultProcessor:
|
||||
self._handle_finish_state_updated_req(
|
||||
req, batch, result, i, logits_output
|
||||
)
|
||||
if track_completions and req.finished():
|
||||
finished_reqs += 1
|
||||
continue
|
||||
|
||||
if (self.enable_overlap or self.enable_overlap_mlx) and (
|
||||
@@ -1048,16 +1041,10 @@ class SchedulerBatchResultProcessor:
|
||||
req.update_finish_state(new_accept_len)
|
||||
|
||||
if sampling_mask_finish_reason is not None:
|
||||
# This path finishes the request too (to_finish ->
|
||||
# finished_reason); count it before the continue.
|
||||
self._handle_sampling_mask_abort(req)
|
||||
if track_completions and req.finished():
|
||||
finished_reqs += 1
|
||||
continue
|
||||
|
||||
self._handle_finish_state_updated_req(req, batch, result, i, logits_output)
|
||||
if track_completions and req.finished():
|
||||
finished_reqs += 1
|
||||
|
||||
if req.return_logprob:
|
||||
self._apply_decode_logprobs(
|
||||
@@ -1094,9 +1081,6 @@ class SchedulerBatchResultProcessor:
|
||||
self._accept_grammar_tokens(req, next_token_id)
|
||||
req.grammar.finished = req.finished()
|
||||
|
||||
if finished_reqs:
|
||||
batch_telemetry.note_request_completed("decode", finished_reqs)
|
||||
|
||||
if auxiliary_output is not None:
|
||||
self.consume_auxiliary_output(
|
||||
batch,
|
||||
|
||||
@@ -21,7 +21,6 @@ from sglang.srt.managers.scheduler_components.metrics_reporter import (
|
||||
from sglang.srt.managers.scheduler_components.recv_skipper import (
|
||||
SchedulerRecvSkipper,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.managers.utils import is_health_check_generate_req
|
||||
from sglang.srt.mem_cache.allocator import BaseTokenToKVPoolAllocator
|
||||
from sglang.srt.mem_cache.base_prefix_cache import BasePrefixCache
|
||||
@@ -33,8 +32,15 @@ from sglang.srt.model_executor.cuda_graph_config import (
|
||||
check_cuda_graph_backend,
|
||||
cuda_graph_fully_disabled,
|
||||
)
|
||||
from sglang.srt.model_executor.encoder_swa_replay import folded_replay_rows
|
||||
from sglang.srt.model_executor.forward_batch_info import ForwardMode
|
||||
from sglang.srt.model_executor.encoder_swa_replay import (
|
||||
decoder_swa_trim_rows,
|
||||
folded_replay_rows,
|
||||
)
|
||||
from sglang.srt.model_executor.forward_batch_info import (
|
||||
CaptureHiddenMode,
|
||||
ForwardMode,
|
||||
get_server_return_hidden_states_mode,
|
||||
)
|
||||
from sglang.srt.model_executor.runner import PrefillCudaGraphRunner
|
||||
from sglang.srt.observability.metrics_collector import (
|
||||
DPBalanceStats,
|
||||
@@ -162,6 +168,8 @@ class MLPSyncBatchInfo:
|
||||
is_health_check: bool = False
|
||||
# Causal (query, key) pairs this rank attends this step.
|
||||
attention_pairs: int = 0
|
||||
# Rows the decoder SWA tail drops before the late layers.
|
||||
decoder_trim_rows: int = 0
|
||||
|
||||
# some gathered elements
|
||||
tp0_info_cpu: torch.Tensor = None
|
||||
@@ -170,6 +178,7 @@ class MLPSyncBatchInfo:
|
||||
global_num_tokens: list[int] = None
|
||||
global_num_tokens_for_logprob: list[int] = None
|
||||
global_attention_pairs: list[int] = None
|
||||
global_decoder_trim_rows: list[int] = None
|
||||
tbo_split_seq_index: torch.Tensor = None
|
||||
global_forward_mode: int = None
|
||||
dp_cooperation_info: Optional[DPCooperationInfo] = None
|
||||
@@ -188,6 +197,7 @@ class MLPSyncBatchInfo:
|
||||
int(self.can_run_draft_cuda_graph),
|
||||
int(self.is_health_check),
|
||||
self.attention_pairs,
|
||||
self.decoder_trim_rows,
|
||||
],
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
@@ -207,6 +217,7 @@ class MLPSyncBatchInfo:
|
||||
1, # can_run_draft_cuda_graph
|
||||
0, # is_health_check
|
||||
0, # attention_pairs
|
||||
0, # decoder_trim_rows
|
||||
],
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
@@ -218,6 +229,7 @@ class MLPSyncBatchInfo:
|
||||
self.global_num_tokens = [self.num_tokens]
|
||||
self.global_num_tokens_for_logprob = [self.num_tokens_for_logprob]
|
||||
self.global_attention_pairs = [self.attention_pairs]
|
||||
self.global_decoder_trim_rows = [self.decoder_trim_rows]
|
||||
self.any_health_check = self.is_health_check
|
||||
if _ENABLE_METRICS_DP_ATTENTION:
|
||||
self.dp_cooperation_info = DPCooperationInfo.create(
|
||||
@@ -295,15 +307,8 @@ class MLPSyncBatchInfo:
|
||||
self.can_run_draft_cuda_graph = bool(tp0_info_cpu[:, 8].min())
|
||||
self.any_health_check = bool(tp0_info_cpu[:, 9].max())
|
||||
self.global_attention_pairs = tp0_info_cpu[:, 10].tolist()
|
||||
self.global_decoder_trim_rows = tp0_info_cpu[:, 11].tolist()
|
||||
self.sync_wait_seconds = time.perf_counter() - sync_start
|
||||
if batch_telemetry.enabled():
|
||||
batch_telemetry.observe_dp_step(
|
||||
self.global_num_tokens,
|
||||
get_parallel().attn_dp_rank,
|
||||
modes=tp0_info_cpu[:, 5].tolist(),
|
||||
health=tp0_info_cpu[:, 9].tolist(),
|
||||
phase="extend" if self.is_extend_in_batch else "decode",
|
||||
)
|
||||
if _ENABLE_METRICS_DP_ATTENTION:
|
||||
self.dp_cooperation_info = DPCooperationInfo.create(
|
||||
tp0_info_cpu[:, 5].tolist()
|
||||
@@ -317,6 +322,7 @@ def _update_gather_batch(
|
||||
draft_require_mlp_tp_gather: Optional[bool] = None,
|
||||
skip_global_metadata=False,
|
||||
):
|
||||
batch.global_decoder_trim_rows = None
|
||||
if not require_mlp_tp_gather:
|
||||
batch.global_num_tokens = [mlp_sync_info.num_tokens]
|
||||
batch.global_num_tokens_for_logprob = [mlp_sync_info.num_tokens_for_logprob]
|
||||
@@ -325,6 +331,12 @@ def _update_gather_batch(
|
||||
batch.global_num_tokens_for_logprob = (
|
||||
mlp_sync_info.global_num_tokens_for_logprob
|
||||
)
|
||||
if (
|
||||
any(mlp_sync_info.global_decoder_trim_rows or ())
|
||||
and min(mlp_sync_info.global_decoder_trim_rows) >= 0
|
||||
):
|
||||
# Every rank resizes its late-layer DP gather when any rank trims.
|
||||
batch.global_decoder_trim_rows = mlp_sync_info.global_decoder_trim_rows
|
||||
# Reuse the same all-gather result for a draft model whose A2A backend
|
||||
# requires a different local/full token-count representation.
|
||||
if draft_require_mlp_tp_gather is not None:
|
||||
@@ -541,6 +553,7 @@ def prepare_mlp_sync_batch_raw(
|
||||
attn_tp_size = parallel.attn_tp_size
|
||||
tp_group = parallel.tp_group
|
||||
# Check if other DP workers have running batches
|
||||
decoder_trim_rows = 0
|
||||
if (
|
||||
local_batch is None
|
||||
or local_batch.forward_mode.is_prebuilt()
|
||||
@@ -558,6 +571,15 @@ def prepare_mlp_sync_batch_raw(
|
||||
and model_runner.attn_backend.folds_encoder_swa_replay
|
||||
):
|
||||
num_tokens += folded_replay_rows(local_batch)
|
||||
if get_exec().features.enable_decoder_swa_bounded_replay:
|
||||
decoder_trim_rows = (
|
||||
-1
|
||||
if get_server_return_hidden_states_mode() == CaptureHiddenMode.FULL
|
||||
else decoder_swa_trim_rows(
|
||||
local_batch,
|
||||
folds_encoder=model_runner.attn_backend.folds_encoder_swa_replay,
|
||||
)
|
||||
)
|
||||
num_tokens_for_logprob = sum(
|
||||
# We should have at least 1 token for sample in every case.
|
||||
max(extend_len - logprob_start_len, 1)
|
||||
@@ -650,6 +672,7 @@ def prepare_mlp_sync_batch_raw(
|
||||
# Gathered unconditionally like the other columns, so a per-process
|
||||
# metrics flag cannot make ranks disagree on what the tensor holds.
|
||||
attention_pairs=_local_attention_pairs(local_batch),
|
||||
decoder_trim_rows=decoder_trim_rows,
|
||||
)
|
||||
|
||||
if num_dp_ranks == 1:
|
||||
@@ -675,15 +698,6 @@ def prepare_mlp_sync_batch_raw(
|
||||
else:
|
||||
need_idle_batch = max(mlp_sync_info.global_num_tokens) > 0
|
||||
|
||||
# Capture the pre-sync local kind before the idle replacement below: the
|
||||
# loop span's local_batch_kind describes what this rank brought in, not the
|
||||
# idle batch it may be asked to run for DP lockstep.
|
||||
_loop_local_kind = (
|
||||
"none"
|
||||
if local_batch is None
|
||||
else ("idle" if local_batch.forward_mode.is_idle() else "extend")
|
||||
)
|
||||
|
||||
batch_to_gather = local_batch
|
||||
if need_idle_batch:
|
||||
if local_batch is None:
|
||||
@@ -692,13 +706,6 @@ def prepare_mlp_sync_batch_raw(
|
||||
# NOTE: for prebuilt batch, we add an inner idle batch to run MLP sync
|
||||
batch_to_gather = local_batch.inner_idle_batch = get_idle_batch()
|
||||
|
||||
_loop_global_outcome = (
|
||||
"idle"
|
||||
if (_loop_local_kind == "none" and need_idle_batch)
|
||||
else ("none" if _loop_local_kind == "none" else _loop_local_kind)
|
||||
)
|
||||
batch_telemetry.note_mlp_sync_outcome(_loop_local_kind, _loop_global_outcome)
|
||||
|
||||
if batch_to_gather is not None:
|
||||
_update_gather_batch(
|
||||
batch_to_gather,
|
||||
@@ -739,12 +746,6 @@ def prepare_mlp_sync_batch_raw(
|
||||
# No batch yet: this iteration may gather again, or end in on_idle.
|
||||
sync_wait_carry.seconds = sync_wait_seconds
|
||||
|
||||
# Stage timing: publish this pass's DP all-gather wait (None = no gather ran).
|
||||
if metadata_ready and not skip_all_gather and num_dp_ranks > 1:
|
||||
batch_telemetry.note_dp_gather(mlp_sync_info.sync_wait_seconds)
|
||||
else:
|
||||
batch_telemetry.note_dp_gather(None)
|
||||
|
||||
return local_batch
|
||||
|
||||
|
||||
|
||||
@@ -46,7 +46,6 @@ class SchedulerLoadInquirer:
|
||||
get_recent_cache_hit_rate: Callable
|
||||
get_stats: Callable
|
||||
get_chunked_req: Callable
|
||||
get_grammar_queue: Callable
|
||||
get_disagg_prefill_bootstrap_queue: Callable
|
||||
get_disagg_prefill_inflight_queue: Callable
|
||||
get_disagg_decode_prealloc_queue: Callable
|
||||
@@ -139,12 +138,6 @@ class SchedulerLoadInquirer:
|
||||
for req in queue
|
||||
)
|
||||
|
||||
# Requests waiting for their grammar to compile reach the queues above
|
||||
# only once it is ready, but they are already this rank's work.
|
||||
grammar_queue = self.get_grammar_queue()
|
||||
waiting_queues.append(grammar_queue)
|
||||
pending_token_queues.append(grammar_queue)
|
||||
|
||||
num_waiting_reqs = sum(len(queue) for queue in waiting_queues)
|
||||
num_used_tokens, kv_token_usage = (
|
||||
self.pool_stats_observer.get_pool_stats().get_kv_token_stats()
|
||||
@@ -152,10 +145,6 @@ class SchedulerLoadInquirer:
|
||||
num_total_tokens = num_used_tokens + sum(
|
||||
req.seqlen for queue in pending_token_queues for req in queue
|
||||
)
|
||||
chunked_req = self.get_chunked_req()
|
||||
if chunked_req is not None and chunked_req.extend_end is not None:
|
||||
# KV covers only the chunks run so far; the rest is still pending.
|
||||
num_total_tokens += max(0, chunked_req.seqlen - chunked_req.extend_end)
|
||||
num_active_tokens = max(0, num_total_tokens - awaiting_kv_tokens)
|
||||
|
||||
memory = None
|
||||
@@ -237,13 +226,6 @@ class SchedulerLoadInquirer:
|
||||
totals = self.get_decode_moment_totals()
|
||||
decode_moments = list(totals) if totals[0] > 0 else None
|
||||
|
||||
_waiting_uncached = self.get_num_waiting_uncached_tokens()
|
||||
_busy_us = self.get_total_prefill_busy_us()
|
||||
_drain_rate = (
|
||||
round(self.get_total_prefill_uncached_tokens() * 1e6 / _busy_us, 2)
|
||||
if _busy_us > 0
|
||||
else 0.0
|
||||
)
|
||||
return LoadSnapshot(
|
||||
dp_rank=int(get_parallel().dp_rank)
|
||||
if get_parallel().dp_rank is not None
|
||||
@@ -251,9 +233,7 @@ class SchedulerLoadInquirer:
|
||||
timestamp=time.time(),
|
||||
num_running_reqs=num_running_reqs,
|
||||
num_waiting_reqs=num_waiting_reqs,
|
||||
num_waiting_uncached_tokens=_waiting_uncached,
|
||||
queued_total_input_tokens=_waiting_uncached,
|
||||
drain_rate_tokens_per_s=_drain_rate,
|
||||
num_waiting_uncached_tokens=self.get_num_waiting_uncached_tokens(),
|
||||
num_used_tokens=num_used_tokens,
|
||||
num_total_tokens=num_total_tokens,
|
||||
num_active_tokens=num_active_tokens,
|
||||
|
||||
@@ -21,7 +21,6 @@ from sglang.srt.observability.forward_pass_metrics import (
|
||||
WelfordAccumulator,
|
||||
wrap_forward_with_fpm,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.metrics_collector import (
|
||||
DPCooperationInfo,
|
||||
QueueCount,
|
||||
@@ -799,21 +798,6 @@ class SchedulerMetricsReporter:
|
||||
self.forward_pass_device_timer._report()
|
||||
msg += ", " + self.forward_occupancy_log_string()
|
||||
|
||||
if batch_telemetry.enabled():
|
||||
_batch_trace_id = batch_telemetry.observe_prefill(
|
||||
dp_rank=get_parallel().attn_dp_rank,
|
||||
iter_no=batch_iter,
|
||||
n_reqs=prefill_stats.num_new_seqs,
|
||||
n_new=prefill_stats.log_input_tokens,
|
||||
n_cached=prefill_stats.log_hit_tokens,
|
||||
queue_depth=len(self.scheduler.waiting_queue),
|
||||
pending_tokens=prefill_stats.num_pending_tokens,
|
||||
duration_s=gap_latency,
|
||||
reqs=batch.reqs if batch is not None else None,
|
||||
)
|
||||
if _batch_trace_id:
|
||||
msg += f", batch-trace: {_batch_trace_id}"
|
||||
|
||||
if self.is_stats_logging_rank:
|
||||
logger.info(msg)
|
||||
if self.current_scheduler_metrics_enabled:
|
||||
@@ -956,20 +940,6 @@ class SchedulerMetricsReporter:
|
||||
self.last_decode_stats_tic = time.perf_counter()
|
||||
self.last_gen_throughput = self.num_generated_tokens / gap_latency
|
||||
|
||||
_decode_batch_trace = None
|
||||
if batch_telemetry.enabled():
|
||||
_decode_batch_trace = batch_telemetry.observe_decode(
|
||||
dp_rank=get_parallel().attn_dp_rank,
|
||||
iter_no=self.forward_ct_decode,
|
||||
n_reqs=len(batch.reqs) if batch is not None else 0,
|
||||
n_new=num_generated_tokens,
|
||||
n_cached=0,
|
||||
queue_depth=len(self.scheduler.waiting_queue),
|
||||
pending_tokens=0,
|
||||
duration_s=gap_latency,
|
||||
reqs=batch.reqs if batch is not None else None,
|
||||
)
|
||||
|
||||
self.num_generated_tokens = 0
|
||||
num_running_reqs = len(batch.reqs)
|
||||
|
||||
@@ -1094,9 +1064,6 @@ class SchedulerMetricsReporter:
|
||||
self.forward_pass_device_timer._report()
|
||||
msg += ", " + self.forward_occupancy_log_string()
|
||||
|
||||
if _decode_batch_trace:
|
||||
msg += f", batch-trace: {_decode_batch_trace}"
|
||||
|
||||
if self.is_stats_logging_rank:
|
||||
logger.info(msg)
|
||||
if self.current_scheduler_metrics_enabled:
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from http import HTTPStatus
|
||||
from typing import (
|
||||
@@ -34,7 +33,6 @@ from sglang.srt.managers.mm_utils import (
|
||||
has_shm_features,
|
||||
unwrap_shm_features,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.observability.scheduler_stage_metrics import (
|
||||
SCHEDULER_STAGE_RECV_REQUESTS,
|
||||
SchedulerStageMetricsRecorder,
|
||||
@@ -95,54 +93,20 @@ class SchedulerRequestReceiver:
|
||||
if not self.recv_skipper.handle(self.get_last_batch()):
|
||||
return []
|
||||
|
||||
_bt = batch_telemetry.recv_timing_enabled()
|
||||
_t0 = time.monotonic() if _bt else 0.0
|
||||
recv_reqs = self._pull_raw_reqs()
|
||||
_t_pull = time.monotonic() if _bt else 0.0
|
||||
|
||||
if self.input_blocker is not None:
|
||||
recv_reqs = self.input_blocker.handle(recv_reqs)
|
||||
|
||||
recv_reqs = self._broadcast_reqs_across_ranks(recv_reqs, local_reqs)
|
||||
_t_bcast = time.monotonic() if _bt else 0.0
|
||||
|
||||
if get_parallel().pp_rank == 0:
|
||||
self.unwrap_pickle_wrapper(recv_reqs)
|
||||
_t_unwrap = time.monotonic() if _bt else 0.0
|
||||
|
||||
recv_reqs = self._apply_mm_receiver(recv_reqs)
|
||||
_t_mm = time.monotonic() if _bt else 0.0
|
||||
|
||||
self._finalize_shm_features(recv_reqs)
|
||||
|
||||
if _bt:
|
||||
n_msgs = 0
|
||||
n_tokens = 0
|
||||
for req in recv_reqs or ():
|
||||
n_msgs += 1
|
||||
ids = getattr(req, "input_ids", None)
|
||||
if ids is None:
|
||||
try:
|
||||
for sub in req:
|
||||
n_tokens += len(getattr(sub, "input_ids", ()) or ())
|
||||
except TypeError:
|
||||
pass
|
||||
else:
|
||||
try:
|
||||
n_tokens += len(ids)
|
||||
except TypeError:
|
||||
pass
|
||||
batch_telemetry.observe_recv(
|
||||
dp_rank=get_parallel().attn_dp_rank,
|
||||
total=time.monotonic() - _t0,
|
||||
pull=_t_pull - _t0,
|
||||
broadcast=_t_bcast - _t_pull,
|
||||
unwrap=_t_unwrap - _t_bcast,
|
||||
mm=_t_mm - _t_unwrap,
|
||||
n_msgs=n_msgs,
|
||||
n_tokens=n_tokens,
|
||||
)
|
||||
|
||||
return recv_reqs
|
||||
|
||||
def _pull_raw_reqs(self) -> Optional[List]:
|
||||
|
||||
@@ -16,7 +16,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import TYPE_CHECKING, List, Optional, Tuple
|
||||
|
||||
@@ -644,21 +643,12 @@ class TpModelWorker(BaseTpWorker):
|
||||
# Replay reads restored main/indexer KV before the normal extend.
|
||||
run_encoder_swa_replay(self, batch)
|
||||
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline
|
||||
|
||||
_ft_t = time.monotonic_ns()
|
||||
except Exception: # pragma: no cover
|
||||
fwd_timeline = None
|
||||
_ft_t = 0
|
||||
forward_batch = ForwardBatch.init_new(
|
||||
batch if folded is None else folded.batch,
|
||||
self.model_runner,
|
||||
capture_hidden_mode=capture_hidden_mode,
|
||||
return_hidden_states_before_norm=False,
|
||||
)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail("fwd_init_new", _ft_t, time.monotonic_ns())
|
||||
if folded is not None:
|
||||
apply_folded_extend(folded, forward_batch)
|
||||
else:
|
||||
|
||||
@@ -24,7 +24,6 @@ from sglang.srt.mem_cache.common import (
|
||||
evict_from_tree_cache,
|
||||
)
|
||||
from sglang.srt.mem_cache.memory_pool import HybridReqToTokenPool, ReqToTokenPool
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.runtime_context import attention_backends, get_parallel
|
||||
from sglang.srt.utils import (
|
||||
is_cpu,
|
||||
@@ -319,10 +318,8 @@ def ensure_mamba_capacity(
|
||||
supports_mamba = tree_cache is not None and tree_cache.supports_mamba()
|
||||
available = allocator.schedulable_available_size()
|
||||
if available < needed and supports_mamba:
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
evict_for_alloc_observed(
|
||||
tree_cache, EvictParams(num_tokens=0, mamba_num=needed - available)
|
||||
tree_cache.evict_for_alloc(
|
||||
EvictParams(num_tokens=0, mamba_num=needed - available)
|
||||
)
|
||||
return allocator.schedulable_available_size() >= needed
|
||||
|
||||
@@ -387,8 +384,7 @@ def alloc_for_extend(
|
||||
# free out-of-window swa tokens
|
||||
batch.maybe_evict_swa()
|
||||
|
||||
with batch_telemetry.detail_phase("prefix_device_indices"):
|
||||
prefix_tensors = [_prefix_kv_indices(batch, r) for r in batch.reqs]
|
||||
prefix_tensors = [_prefix_kv_indices(batch, r) for r in batch.reqs]
|
||||
|
||||
reuse_kv = None
|
||||
if batch.is_dllm():
|
||||
|
||||
@@ -165,11 +165,7 @@ class BaseTokenToKVPoolAllocator(abc.ABC):
|
||||
return
|
||||
shortfall = num_tokens - self.available_size()
|
||||
if shortfall > 0:
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
evict_for_alloc_observed(
|
||||
tree_cache, EvictParams(num_tokens=shortfall)
|
||||
)
|
||||
tree_cache.evict_for_alloc(EvictParams(num_tokens=shortfall))
|
||||
_evict_until_allocatable(tree_cache, self, num_tokens)
|
||||
|
||||
def check_decode_capacity(
|
||||
|
||||
@@ -189,12 +189,10 @@ class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
|
||||
"""
|
||||
from sglang.srt.mem_cache.base_prefix_cache import EvictParams
|
||||
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
available = self.swa_available_size()
|
||||
if available < swa_tokens:
|
||||
evict_for_alloc_observed(
|
||||
tree_cache, EvictParams(swa_num_tokens=swa_tokens - available)
|
||||
tree_cache.evict_for_alloc(
|
||||
EvictParams(swa_num_tokens=swa_tokens - available)
|
||||
)
|
||||
available = self.swa_available_size()
|
||||
if available < swa_tokens:
|
||||
@@ -217,11 +215,8 @@ class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
|
||||
full_shortfall = max(0, num_tokens - self.full_available_size())
|
||||
swa_shortfall = max(0, num_tokens - self.swa_available_size())
|
||||
if full_shortfall or swa_shortfall:
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
evict_for_alloc_observed(
|
||||
tree_cache,
|
||||
EvictParams(num_tokens=full_shortfall, swa_num_tokens=swa_shortfall),
|
||||
tree_cache.evict_for_alloc(
|
||||
EvictParams(num_tokens=full_shortfall, swa_num_tokens=swa_shortfall)
|
||||
)
|
||||
|
||||
def full_available_size(self):
|
||||
|
||||
@@ -1254,10 +1254,7 @@ class UnifiedSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
|
||||
return
|
||||
full_reclaim, swa_reclaim = reclaim_plan
|
||||
if full_reclaim or swa_reclaim:
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
evict_for_alloc_observed(
|
||||
tree_cache,
|
||||
tree_cache.evict_for_alloc(
|
||||
EvictParams(num_tokens=full_reclaim, swa_num_tokens=swa_reclaim),
|
||||
allocation_reclaim_satisfied=lambda: (
|
||||
self.reclaim_plan(num_tokens, required_swa) == (0, 0)
|
||||
@@ -1912,10 +1909,7 @@ class UnifiedMambaSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
|
||||
# precedes state reclaim so Mamba-first tree ordering cannot skip it.
|
||||
state_freed = 0
|
||||
if full_reclaim or swa_reclaim:
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
result = evict_for_alloc_observed(
|
||||
tree_cache,
|
||||
result = tree_cache.evict_for_alloc(
|
||||
EvictParams(num_tokens=num_tokens, swa_num_tokens=num_tokens),
|
||||
allocation_reclaim_satisfied=lambda: self._token_reclaim_satisfied(
|
||||
num_tokens
|
||||
@@ -1936,10 +1930,7 @@ class UnifiedMambaSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
|
||||
)
|
||||
state_quota = max(0, -(-pair_bytes // ma.entry_bytes) - state_freed)
|
||||
if state_quota and tree_cache.mamba_evictable_size():
|
||||
from sglang.srt.mem_cache.common import evict_for_alloc_observed
|
||||
|
||||
result = evict_for_alloc_observed(
|
||||
tree_cache,
|
||||
result = tree_cache.evict_for_alloc(
|
||||
EvictParams(mamba_num=state_quota),
|
||||
allocation_reclaim_satisfied=lambda: self._token_reclaim_satisfied(
|
||||
num_tokens
|
||||
|
||||
@@ -25,7 +25,6 @@ from __future__ import annotations
|
||||
import inspect
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from typing import (
|
||||
Callable,
|
||||
Dict,
|
||||
@@ -441,9 +440,6 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
|
||||
# site; `_flush` materializes the write-set lazily, avoiding a sync here.
|
||||
self._inflight_forward: Optional[Tuple[torch.cuda.Event, torch.Tensor]] = None
|
||||
self._hicache_transfer_done_events: Dict[Hashable, torch.cuda.Event] = {}
|
||||
# The async backup submit worker records transfer events while the
|
||||
# scheduler may concurrently iterate/clear them for compaction gating.
|
||||
self._hicache_transfer_done_events_lock = threading.Lock()
|
||||
# HiCache reserves SWA physical pages during load-back, then binds them
|
||||
# into the tree before the H2D copy is submitted. Compaction must not
|
||||
# move those pages in that interval because the queued PoolTransfer still
|
||||
@@ -561,8 +557,7 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
|
||||
self.live_page_count = 0
|
||||
self._inflight_forward = None
|
||||
self._latest_forward_done_event = None
|
||||
with self._hicache_transfer_done_events_lock:
|
||||
self._hicache_transfer_done_events.clear()
|
||||
self._hicache_transfer_done_events.clear()
|
||||
self._pending_hicache_load_pages = 0
|
||||
|
||||
def clear_inverse_history(self) -> None:
|
||||
@@ -2088,8 +2083,7 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
|
||||
_pending_hicache_load_pages: _CapacityField[int] = _CapacityField()
|
||||
|
||||
def set_hicache_transfer_done_event(self, transfer_key: Hashable, event) -> None:
|
||||
with self._hicache_transfer_done_events_lock:
|
||||
self._hicache_transfer_done_events[transfer_key] = event
|
||||
self._hicache_transfer_done_events[transfer_key] = event
|
||||
if transfer_key == "load" or (
|
||||
isinstance(transfer_key, tuple) and transfer_key[-1] == "load"
|
||||
):
|
||||
@@ -2101,12 +2095,11 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
|
||||
"""Order the current stream after outstanding HiCache copies. Only moves
|
||||
and frees need this: a copy's source and target are not free pages until
|
||||
its finish event is observed, so allocation never reuses them."""
|
||||
with self._hicache_transfer_done_events_lock:
|
||||
if not self._hicache_transfer_done_events:
|
||||
return
|
||||
events = tuple(self._hicache_transfer_done_events.values())
|
||||
self._hicache_transfer_done_events.clear()
|
||||
if not self._hicache_transfer_done_events:
|
||||
return
|
||||
current_stream = torch.cuda.current_stream()
|
||||
events = tuple(self._hicache_transfer_done_events.values())
|
||||
self._hicache_transfer_done_events.clear()
|
||||
for event in events:
|
||||
current_stream.wait_event(event)
|
||||
|
||||
|
||||
@@ -24,7 +24,6 @@ from sglang.srt.mem_cache.hicache_storage import PoolTransfer
|
||||
from sglang.srt.mem_cache.memory_pool import ReqToTokenPool
|
||||
from sglang.srt.mem_cache.radix_cache import RadixKey
|
||||
from sglang.srt.mem_cache.unified_cache.component_type import ComponentType
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.runtime_context import get_serving, get_spec
|
||||
from sglang.srt.utils.common import ceil_align
|
||||
|
||||
@@ -48,22 +47,6 @@ def kv_to_page_indices(kv_indices: torch.Tensor, page_size: int) -> np.ndarray:
|
||||
return (kv_indices[::page_size] // page_size).cpu().numpy()
|
||||
|
||||
|
||||
def kv_to_page_indices_pinned(
|
||||
kv_indices: torch.Tensor, page_size: int
|
||||
) -> torch.Tensor:
|
||||
"""Same values/dtype as :func:`kv_to_page_indices`, in a pinned CPU tensor.
|
||||
|
||||
The device-to-host copy is non-blocking on the current stream. The caller
|
||||
must record a CUDA event on that stream after all such copies and hand it
|
||||
to the transfer worker, which materializes the tensor only after the event
|
||||
(see ``materialize_state_indices``).
|
||||
"""
|
||||
src = kv_indices[::page_size] // page_size
|
||||
out = torch.empty(src.shape, dtype=src.dtype, device="cpu", pin_memory=True)
|
||||
out.copy_(src, non_blocking=True)
|
||||
return out
|
||||
|
||||
|
||||
def suffix_page_indices(
|
||||
req_to_token: torch.Tensor,
|
||||
req_pool_idx: int,
|
||||
@@ -285,17 +268,6 @@ def match_kv_cache(
|
||||
max_len = req._compute_max_prefix_len(len(token_ids))
|
||||
req.num_matched_prefix_tokens = min(req.prefix_len + req.host_hit_length, max_len)
|
||||
req.swa_branching_seqlen = match_result.swa_branching_seqlen
|
||||
# Observability: record the Full-KV length the components could have used.
|
||||
# This function also runs on waiting-queue probes, PD decode and HiCache
|
||||
# rematches, so it only stores the gap; the metric is emitted exactly once
|
||||
# per prefill admission in PrefillAdder._commit_prefill_admission, where the
|
||||
# accepted prefix length is final (after any host load-back).
|
||||
req.full_kv_hit_length = match_result.full_kv_hit_length
|
||||
req.swa_gate_reject_tokens = max(
|
||||
0,
|
||||
match_result.full_kv_hit_length
|
||||
- (match_result.device_prefix_len + match_result.host_hit_length),
|
||||
)
|
||||
# A probe match keeps what it did not report; a new round resets both.
|
||||
if match_result.mamba_branching_seqlen is not None:
|
||||
req.mamba_branching_seqlen = match_result.mamba_branching_seqlen
|
||||
@@ -316,86 +288,13 @@ def checkpoint_kv_cache(req: Req, tree_cache: BasePrefixCache) -> None:
|
||||
req.prefix_len = req.extend_end
|
||||
|
||||
|
||||
def _rank_consensus_enabled() -> bool:
|
||||
"""Whether the rank-consensus checker is active (cheap, best effort)."""
|
||||
try:
|
||||
from sglang.srt.utils import rank_consensus_checker
|
||||
|
||||
return bool(rank_consensus_checker.enabled())
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def evict_for_alloc_observed(tree_cache, params, **kwargs):
|
||||
"""Time ``tree_cache.evict_for_alloc`` and record what it realized.
|
||||
|
||||
Kept at the call site (not inside the callee) so decorator/consensus waits
|
||||
are part of the measurement. ``kwargs`` forwards the unified allocators'
|
||||
``allocation_reclaim_satisfied`` predicate.
|
||||
"""
|
||||
with batch_telemetry.detail_phase("evict_for_alloc"):
|
||||
result = tree_cache.evict_for_alloc(params, **kwargs)
|
||||
if batch_telemetry.loop_detail_enabled() and result is not None:
|
||||
batch_telemetry.detail_count("evict_for_alloc_calls")
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_plan_requested_tokens", int(getattr(params, "num_tokens", 0) or 0)
|
||||
)
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_plan_requested_swa_tokens",
|
||||
int(getattr(params, "swa_num_tokens", 0) or 0),
|
||||
)
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_plan_requested_mamba",
|
||||
int(getattr(params, "mamba_num", 0) or 0),
|
||||
)
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_evicted_tokens",
|
||||
int(getattr(result, "num_tokens_evicted", 0) or 0),
|
||||
)
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_evicted_swa_tokens",
|
||||
int(getattr(result, "swa_num_tokens_evicted", 0) or 0),
|
||||
)
|
||||
# Record the realized Mamba count unconditionally (including zero) so
|
||||
# "no Mamba demand" and "demand but nothing reclaimed" stay distinct.
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_evicted_mamba",
|
||||
int(getattr(result, "mamba_num_evicted", 0) or 0),
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def evict_from_tree_cache(
|
||||
tree_cache: BasePrefixCache | None, num_tokens: int
|
||||
) -> bool | None:
|
||||
if tree_cache is not None and tree_cache.supports_prefix_sharing():
|
||||
allocator = tree_cache.token_to_kv_pool_allocator
|
||||
with batch_telemetry.detail_phase("evict_to_free_tokens"):
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
available_before = allocator.available_size()
|
||||
batch_telemetry.detail_count("evict_calls")
|
||||
# Demand is what the batch wants allocatable; shortfall is the
|
||||
# part that eviction must actually recover. Keep them apart:
|
||||
# summing demand as "requested eviction" overstates eviction.
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_demand_tokens", int(num_tokens)
|
||||
)
|
||||
batch_telemetry.detail_attr_sum(
|
||||
"evict_shortfall_tokens",
|
||||
max(0, int(num_tokens) - int(available_before)),
|
||||
)
|
||||
batch_telemetry.detail_attr_setdefault(
|
||||
"evict_free_tokens_before", int(available_before)
|
||||
)
|
||||
batch_telemetry.detail_attr(
|
||||
"evict_rank_consensus_checker", _rank_consensus_enabled()
|
||||
)
|
||||
result = allocator.evict_to_free_tokens(tree_cache, num_tokens)
|
||||
if batch_telemetry.loop_detail_enabled():
|
||||
batch_telemetry.detail_attr(
|
||||
"evict_free_tokens_after", int(allocator.available_size())
|
||||
)
|
||||
return result
|
||||
return tree_cache.token_to_kv_pool_allocator.evict_to_free_tokens(
|
||||
tree_cache, num_tokens
|
||||
)
|
||||
|
||||
|
||||
def _evict_until_allocatable(
|
||||
|
||||
@@ -41,7 +41,6 @@ from sglang.srt.mem_cache.pool_host.base import uses_shared_host_layout
|
||||
from sglang.srt.mem_cache.pool_host.mha import MHATokenToKVPoolHost
|
||||
from sglang.srt.mem_cache.pool_host.unified import UnifiedPageEnvelopeHostPool
|
||||
from sglang.srt.mem_cache.radix_cache import RadixKey
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.srt.mem_cache.allocator import BaseTokenToKVPoolAllocator
|
||||
@@ -556,17 +555,15 @@ class HybridCacheController(BaseHiCacheController):
|
||||
return None
|
||||
host_indices, pool_transfers = allocation
|
||||
|
||||
op = CacheOperation(
|
||||
host_indices,
|
||||
device_indices,
|
||||
node_id,
|
||||
priority,
|
||||
pool_transfers=pool_transfers or None,
|
||||
self.write_queue.append(
|
||||
CacheOperation(
|
||||
host_indices,
|
||||
device_indices,
|
||||
node_id,
|
||||
priority,
|
||||
pool_transfers=pool_transfers or None,
|
||||
)
|
||||
)
|
||||
# Start the pinned index D2H before the (possibly deferred) flush, so
|
||||
# the copy can run while the scheduler keeps going.
|
||||
self._start_async_index_copy(op)
|
||||
self.write_queue.append(op)
|
||||
if flush:
|
||||
self.start_writing()
|
||||
return host_indices
|
||||
@@ -1512,30 +1509,26 @@ class HybridCacheController(BaseHiCacheController):
|
||||
self, host_pool, host_indices, device_indices, *, write_back_jit: bool
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
if uses_shared_host_layout(host_pool):
|
||||
batch_telemetry.flush_count("shared_layout_pools")
|
||||
return host_indices, device_indices
|
||||
if write_back_jit and host_pool.can_use_write_back_jit:
|
||||
if host_indices.is_cuda:
|
||||
with batch_telemetry.backup_phase("host_indices_cpu"):
|
||||
host_indices = host_indices.cpu()
|
||||
host_indices = host_indices.cpu()
|
||||
return host_indices, device_indices
|
||||
return self.move_indices(host_indices, device_indices)
|
||||
|
||||
def move_hybrid_indices(
|
||||
self, operation: CacheOperation, *, write_back_jit: bool = False
|
||||
) -> tuple[torch.Tensor, torch.Tensor, Optional[list[PoolTransfer]]]:
|
||||
with batch_telemetry.backup_phase("anchor_indices"):
|
||||
host_indices, device_indices = self._move_pool_indices(
|
||||
self.mem_pool_host.anchor_entry.host_pool,
|
||||
operation.host_indices,
|
||||
operation.device_indices,
|
||||
write_back_jit=write_back_jit,
|
||||
)
|
||||
host_indices, device_indices = self._move_pool_indices(
|
||||
self.mem_pool_host.anchor_entry.host_pool,
|
||||
operation.host_indices,
|
||||
operation.device_indices,
|
||||
write_back_jit=write_back_jit,
|
||||
)
|
||||
resolved_pool_transfers = None
|
||||
if operation.pool_transfers:
|
||||
resolved_pool_transfers = []
|
||||
for transfer in operation.pool_transfers:
|
||||
_pool_t0 = batch_telemetry.flush_mark()
|
||||
transfer_host_indices, transfer_device_indices = (
|
||||
self._move_pool_indices(
|
||||
self.mem_pool_host.entry_map[transfer.name].host_pool,
|
||||
@@ -1544,9 +1537,6 @@ class HybridCacheController(BaseHiCacheController):
|
||||
write_back_jit=write_back_jit,
|
||||
)
|
||||
)
|
||||
batch_telemetry.flush_phase_since(
|
||||
"sidecar_indices." + str(transfer.name.value), _pool_t0
|
||||
)
|
||||
# Tree-owned transfers keep their original radix-tree indices.
|
||||
resolved_pool_transfers.append(
|
||||
replace(
|
||||
@@ -1557,31 +1547,19 @@ class HybridCacheController(BaseHiCacheController):
|
||||
)
|
||||
return host_indices, device_indices, resolved_pool_transfers
|
||||
|
||||
def _page_transfer(
|
||||
self,
|
||||
operation: PrefetchOperation,
|
||||
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
|
||||
) -> int:
|
||||
def _page_transfer(self, operation: PrefetchOperation) -> bool:
|
||||
with self.mem_pool_host.layout_lease():
|
||||
return self._page_transfer_with_stable_layout(operation, ack_sink)
|
||||
return self._page_transfer_with_stable_layout(operation)
|
||||
|
||||
def _page_transfer_with_stable_layout(
|
||||
self,
|
||||
operation: PrefetchOperation,
|
||||
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
|
||||
) -> int:
|
||||
def _page_transfer_with_stable_layout(self, operation: PrefetchOperation) -> None:
|
||||
# KV pools and KV-derived pools first — determines actual completed page count
|
||||
kv_completed_pages = super()._page_transfer(operation, ack_sink)
|
||||
kv_completed_pages = super()._page_transfer(operation)
|
||||
|
||||
# Read non-KV derived sidecar pool, e.g. SWA, Mamba.
|
||||
self._page_transfer_sidecar(operation, kv_completed_pages, ack_sink)
|
||||
return kv_completed_pages
|
||||
self._page_transfer_sidecar(operation, kv_completed_pages)
|
||||
|
||||
def _page_transfer_sidecar(
|
||||
self,
|
||||
operation: PrefetchOperation,
|
||||
kv_completed_pages: int,
|
||||
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
|
||||
self, operation: PrefetchOperation, kv_completed_pages: int
|
||||
) -> None:
|
||||
if operation.pool_transfers is None:
|
||||
return
|
||||
@@ -1626,19 +1604,17 @@ class HybridCacheController(BaseHiCacheController):
|
||||
"HiCache sidecar prefetch failed for request %s",
|
||||
operation.request_id,
|
||||
)
|
||||
# Emit PrefetchAck even when the operation has been canceled by the
|
||||
# scheduler thread. The sync thread expects the same number of ACKs on
|
||||
# every rank. Pipelined workers buffer this ACK per operation instead of
|
||||
# publishing it directly to the shared queue.
|
||||
ack = PrefetchAck(
|
||||
rid=operation.request_id,
|
||||
operation=operation,
|
||||
pool_hits=pool_hits,
|
||||
# Emit PrefetchAck to prefetch_sync_queue, even the operation has been canceled by the
|
||||
# scheduler thread. The prefetch sync thread expects the same number of PrefetchAck objects
|
||||
# to perform all_reduce.
|
||||
self.prefetch_sync_queue.put(
|
||||
PrefetchAck(
|
||||
rid=operation.request_id,
|
||||
operation=operation,
|
||||
pool_hits=pool_hits,
|
||||
)
|
||||
)
|
||||
if ack_sink is None:
|
||||
self.prefetch_sync_queue.put(ack)
|
||||
else:
|
||||
ack_sink(ack)
|
||||
return
|
||||
|
||||
def trim_prefetch_full_head(self, operation, trim_tokens: int) -> None:
|
||||
"""Drop a device-resident FULL head after the availability query;
|
||||
@@ -1728,15 +1704,22 @@ class HybridCacheController(BaseHiCacheController):
|
||||
|
||||
return False
|
||||
|
||||
def _should_skip_backup(self) -> bool:
|
||||
# Replicated primary MLA KV may be TP0-only, but rank-sharded sidecars
|
||||
# still require IO on every rank. _page_backup applies should_backup()
|
||||
# to each individual transfer, so the whole operation cannot be skipped.
|
||||
return False
|
||||
|
||||
def backup_thread_func(self):
|
||||
"""Use the base pipeline while retaining per-pool writer selection."""
|
||||
super().backup_thread_func()
|
||||
"""Back up rank-sharded sidecars on every TP rank.
|
||||
|
||||
The base implementation skips the entire operation on non-zero MLA TP
|
||||
ranks. That optimization is valid for replicated MLA KV, but not for
|
||||
hybrid rank-sharded pools such as Kimi-K3 Mamba state.
|
||||
"""
|
||||
while not self.storage_stop_event.is_set():
|
||||
try:
|
||||
operation = self.backup_queue.get(block=True, timeout=1)
|
||||
if operation is None:
|
||||
continue
|
||||
self._page_backup_or_log(operation)
|
||||
self.ack_backup_queue.put(operation)
|
||||
except Empty:
|
||||
continue
|
||||
|
||||
def _resolve_sidecar_kv_derived_pool_transfers(self, operation):
|
||||
for transfer in operation.pool_transfers:
|
||||
|
||||
@@ -9,7 +9,6 @@ import torch
|
||||
|
||||
from sglang.srt.environ import envs
|
||||
from sglang.srt.mem_cache.pool_host.base import shared_host_layout_domains
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.utils import get_device_module
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -72,13 +71,10 @@ class L2TransferEngine:
|
||||
original_device = transfer.device_indices.device
|
||||
# The direct backend supplies CPU indices even for a CUDA pool.
|
||||
# Translate alongside the v2p table, then restore the backend's device.
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
indices = transfer.device_indices.to(
|
||||
getattr(transfer.device_pool, "device", original_device)
|
||||
).contiguous()
|
||||
batch_telemetry.flush_phase_since("resolve_indices.to_device", _t0)
|
||||
dcp_size = getattr(transfer.host_pool, "dcp_size", 1)
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
if dcp_size > 1:
|
||||
# The MLA host pool selects this rank and collapses logical IDs.
|
||||
# Translate in local virtual space, then preserve that widened
|
||||
@@ -86,22 +82,16 @@ class L2TransferEngine:
|
||||
resolved = translate(indices // dcp_size) * dcp_size + indices % dcp_size
|
||||
else:
|
||||
resolved = translate(indices)
|
||||
batch_telemetry.flush_phase_since("resolve_indices.translate", _t0)
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
result = resolved.to(original_device)
|
||||
batch_telemetry.flush_phase_since("resolve_indices.back_to_host", _t0)
|
||||
return result
|
||||
return resolved.to(original_device)
|
||||
|
||||
def _prepare_transfers(self, transfers: list[L2Transfer]) -> list[L2Transfer]:
|
||||
prepared = []
|
||||
for transfer in transfers:
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
host_indices, device_indices = transfer.host_pool.prepare_transfer_indices(
|
||||
transfer.host_indices,
|
||||
self._resolve_device_indices(transfer),
|
||||
self.io_backend,
|
||||
)
|
||||
batch_telemetry.flush_phase_since("prepare_transfer_indices", _t0)
|
||||
prepared.append(
|
||||
transfer._replace(
|
||||
host_indices=host_indices,
|
||||
@@ -112,55 +102,44 @@ class L2TransferEngine:
|
||||
|
||||
@contextmanager
|
||||
def _submission(self, transfers, stream, transfer_key, start_event=None):
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
start_event = self._start_event(start_event)
|
||||
ack_start, ack_finish, timing_enabled = make_timing_event_pair()
|
||||
completion = TransferCompletion(ack_start, ack_finish, timing_enabled)
|
||||
domains = shared_host_layout_domains(t.host_pool for t in transfers)
|
||||
for domain in domains:
|
||||
domain.acquire_layout()
|
||||
batch_telemetry.flush_phase_since("submit_setup", _t0)
|
||||
finish_recorded = False
|
||||
try:
|
||||
with device_module.stream(stream):
|
||||
# Index preparation can read producer-owned device indices.
|
||||
with batch_telemetry.backup_phase("start_event_wait"):
|
||||
start_event.wait(stream)
|
||||
with batch_telemetry.backup_phase("prepare_transfers"):
|
||||
prepared = self._prepare_transfers(transfers)
|
||||
with batch_telemetry.backup_phase("ack_events"):
|
||||
ack_start.record()
|
||||
start_event.wait(stream)
|
||||
prepared = self._prepare_transfers(transfers)
|
||||
ack_start.record()
|
||||
yield prepared, completion
|
||||
with batch_telemetry.backup_phase("ack_events"):
|
||||
ack_finish.record()
|
||||
finish_recorded = True
|
||||
self._record_stream(transfers + prepared, stream)
|
||||
ack_finish.record()
|
||||
finish_recorded = True
|
||||
self._record_stream(transfers + prepared, stream)
|
||||
except Exception:
|
||||
stream.synchronize()
|
||||
raise
|
||||
finally:
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
for domain in reversed(domains):
|
||||
domain.release_layout(
|
||||
ack_finish if finish_recorded else None,
|
||||
(id(self), transfer_key),
|
||||
)
|
||||
batch_telemetry.flush_phase_since("layout_release", _t0)
|
||||
|
||||
def submit_device_to_host(self, transfers: list[L2Transfer]) -> TransferCompletion:
|
||||
with self._submission(
|
||||
transfers, self.device_to_host_stream, "device_to_host"
|
||||
) as (transfers, completion):
|
||||
for transfer in transfers:
|
||||
_t0 = batch_telemetry.flush_mark()
|
||||
transfer.host_pool.backup_from_device_all_layer_physical(
|
||||
transfer.device_pool,
|
||||
transfer.host_indices,
|
||||
transfer.device_indices,
|
||||
self.io_backend,
|
||||
)
|
||||
batch_telemetry.flush_phase_since("launch_pool", _t0)
|
||||
batch_telemetry.flush_count("launch_pools")
|
||||
return completion
|
||||
|
||||
def submit_host_to_device(
|
||||
|
||||
@@ -526,22 +526,6 @@ class MooncakeStore(HiCacheStorage, MooncakeBaseStore):
|
||||
transfer_engine = None
|
||||
|
||||
setup_kwargs = {}
|
||||
# Opt-in client metrics HTTP server (Prometheus). Each rank
|
||||
# embeds its own Mooncake client, so give every rank its own
|
||||
# port: base + dp_rank. Off unless the env var is set.
|
||||
_mc_metrics_port_base = os.environ.get(
|
||||
"SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE"
|
||||
)
|
||||
if _mc_metrics_port_base:
|
||||
_mc_dp_rank = (
|
||||
int(getattr(storage_config, "dp_rank", 0) or 0)
|
||||
if storage_config is not None
|
||||
else 0
|
||||
)
|
||||
setup_kwargs["enable_client_http_server"] = True
|
||||
setup_kwargs["client_http_port"] = (
|
||||
int(_mc_metrics_port_base) + _mc_dp_rank
|
||||
)
|
||||
if self.config.enable_ssd_offload:
|
||||
setup_kwargs["enable_ssd_offload"] = True
|
||||
if self.config.ssd_offload_path is not None:
|
||||
|
||||
@@ -111,13 +111,11 @@ from sglang.srt.observability.metrics_collector import (
|
||||
StorageMetrics,
|
||||
StorageMetricsCollector,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
from sglang.srt.runtime_context import (
|
||||
get_disagg,
|
||||
get_memory,
|
||||
get_model,
|
||||
get_observability,
|
||||
get_parallel,
|
||||
)
|
||||
from sglang.srt.session.streaming_session import StreamingSession, is_virtual_node
|
||||
from sglang.srt.utils.common import ceil_align
|
||||
@@ -439,8 +437,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
CacheRequestHandle, int
|
||||
] = {}
|
||||
self.storage_prefetch_retries = StoragePrefetchRetries()
|
||||
# request -> monotonic stamp of the first in-flight check (ready latency)
|
||||
self._prefetch_issue_ts: dict[CacheRequestHandle, float] = {}
|
||||
self.ongoing_backup: dict[int, tuple[NodeId, DecLockRefParams]] = {}
|
||||
if self.buffer_pipeline is not None:
|
||||
self.buffer_pipeline.reset()
|
||||
@@ -1694,16 +1690,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
# backed up. Skip only when no transfer remains.
|
||||
if device_value.numel() == 0 and not comp_xfers:
|
||||
continue
|
||||
# A split fragment keeps the original node's ack, and component
|
||||
# transfers can cover other nodes. Check the whole publish set,
|
||||
# not just ack keys, before assigning any node a new ack owner.
|
||||
publish_set = set(self._backup_publish_node_ids(node_id, comp_xfers))
|
||||
# Write-back eviction has its own round-wide completion barrier.
|
||||
if not write_back and any(
|
||||
not publish_set.isdisjoint(pending.publish_node_ids)
|
||||
for pending in self.ongoing_write_through.values()
|
||||
):
|
||||
self.writing_check(write_back=True)
|
||||
sidecar_xfers = self._build_backup_sidecar(device_value, comp_xfers)
|
||||
host_indices = self._execute_kv_backup(
|
||||
node_id, device_value, comp_xfers, sidecar_xfers
|
||||
@@ -2242,8 +2228,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
anchor_lock_params,
|
||||
comp_xfers,
|
||||
)
|
||||
if batch_telemetry.stage_timing_enabled():
|
||||
self._prefetch_issue_ts.setdefault(request, time.monotonic())
|
||||
if buffer_mode:
|
||||
# The query reads no tree state; pinning across it would hold the
|
||||
# anchor cap for a storage round trip, so the pin waits for IO commit.
|
||||
@@ -2421,22 +2405,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
self._handle_prefetch_result(operation)
|
||||
return True
|
||||
|
||||
def _observe_prefetch_ready(
|
||||
self, request: CacheRequestHandle, hit: bool
|
||||
) -> None:
|
||||
started = self._prefetch_issue_ts.pop(request, None)
|
||||
if started is None:
|
||||
return
|
||||
try:
|
||||
dp_rank = get_parallel().attn_dp_rank
|
||||
except Exception:
|
||||
dp_rank = 0
|
||||
batch_telemetry.observe_prefetch_ready(
|
||||
dp_rank=dp_rank,
|
||||
result="hit" if hit else "miss",
|
||||
seconds=time.monotonic() - started,
|
||||
)
|
||||
|
||||
def _handle_prefetch_result(self, operation: PrefetchOperation) -> None:
|
||||
# This function **owns**:
|
||||
# - host_indices[0 : completed_tokens]
|
||||
@@ -2531,9 +2499,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
|
||||
self.dec_host_lock_ref(last_host_node_id, anchor_lock_params)
|
||||
del self.ongoing_prefetch[request]
|
||||
# Success path retires here instead of _retire_ongoing_prefetch(); the
|
||||
# ready-latency metric must cover it or every L3 hit goes unmeasured.
|
||||
self._observe_prefetch_ready(request, loaded_from_storage > 0)
|
||||
self.cache_controller.prefetch_tokens_occupied -= len(prefetch_key)
|
||||
|
||||
self.prefetch_loaded_tokens_by_reqid[request] = loaded_from_storage
|
||||
@@ -2788,7 +2753,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
|
||||
@rank_consensus(same_params=True)
|
||||
def release_aborted_request(self, request: CacheRequestHandle) -> None:
|
||||
self._prefetch_issue_ts.pop(request, None)
|
||||
rid = request.rid
|
||||
self._storage_prefetch_wait.pop(request, None)
|
||||
if self.linker is not None:
|
||||
@@ -2836,9 +2800,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
) -> None:
|
||||
"""Drop an in-flight prefetch, releasing ``host_indices`` and ``extra_pools``."""
|
||||
info = self.ongoing_prefetch.pop(request)
|
||||
self._observe_prefetch_ready(
|
||||
request, bool(info.operation.storage_hit_count)
|
||||
)
|
||||
if info.anchor_lock_params is not None:
|
||||
self.dec_host_lock_ref(info.anchor_node_id, info.anchor_lock_params)
|
||||
if self.buffer_pipeline is not None:
|
||||
@@ -3505,13 +3466,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
if write_back:
|
||||
# Blocking: submit what is still queued, then wait for every ack.
|
||||
cc.start_writing()
|
||||
# Async submissions cover ACKs asynchronously; wait for them before
|
||||
# asserting that every write-through node has been acknowledged.
|
||||
if hasattr(cc, "wait_write_submissions_idle"):
|
||||
cc.wait_write_submissions_idle()
|
||||
_drain = getattr(cc, "drain_write_submission_results", None)
|
||||
if _drain is not None:
|
||||
_drain()
|
||||
while self.ongoing_write_through:
|
||||
for ack in cc.ack_write_queue:
|
||||
ack.finish_event.synchronize()
|
||||
@@ -3524,9 +3478,6 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
return
|
||||
|
||||
if finish_count is None:
|
||||
_drain = getattr(cc, "drain_write_submission_results", None)
|
||||
if _drain is not None:
|
||||
_drain()
|
||||
# Every rank must enter the all_reduce below; ongoing_write_through can
|
||||
# diverge across ranks (e.g. write_backup returning 0 on a subset).
|
||||
finish_count = 0
|
||||
@@ -3704,16 +3655,9 @@ class UnifiedRadixCache(BasePrefixCache):
|
||||
|
||||
# Reap the previous round's PP-sync sends before issuing new ones.
|
||||
self._drain_async_work()
|
||||
# Collect ACKs finished by the async submit worker before counting them.
|
||||
# getattr keeps lightweight controller fakes (unit tests) working.
|
||||
_drain = getattr(self.cache_controller, "drain_write_submission_results", None)
|
||||
if _drain is not None:
|
||||
_drain()
|
||||
# Backups queued outside process_batch_result: the chunked-prefill stash
|
||||
# in get_next_batch_to_run, abort_request, and the PD prefill release.
|
||||
self.flush_pending_backups()
|
||||
if _drain is not None:
|
||||
_drain()
|
||||
|
||||
(
|
||||
write_finish_count,
|
||||
|
||||
@@ -52,6 +52,23 @@ def folded_replay_rows(batch) -> int:
|
||||
)
|
||||
|
||||
|
||||
def decoder_swa_trim_rows(batch, *, folds_encoder: bool) -> int:
|
||||
"""Rows removed by a plain extend's decoder tail; -1 requests the backend
|
||||
exchange when prompt outputs require whole-extend rows."""
|
||||
if not batch.forward_mode.is_extend_without_speculative():
|
||||
return 0
|
||||
if batch.return_logprob or batch.return_hidden_states:
|
||||
return -1
|
||||
lens = list(batch.extend_lens)
|
||||
if folds_encoder and batch.encoder_swa_reset is not None:
|
||||
for i, (reset, prefix) in enumerate(
|
||||
zip(batch.encoder_swa_reset, batch.prefix_lens)
|
||||
):
|
||||
if reset:
|
||||
lens[i] += replay_len(prefix)
|
||||
return sum(max(0, n - 128) for n in lens)
|
||||
|
||||
|
||||
def apply_folded_extend(folded: FoldedExtend, forward_batch) -> None:
|
||||
forward_batch.encoder_swa_row_floor = folded.row_floor
|
||||
forward_batch.encoder_swa_compress_rows = folded.keep_rows
|
||||
|
||||
@@ -51,6 +51,7 @@ from sglang.srt.layers.dp_attention import (
|
||||
set_is_extend_in_batch,
|
||||
world_dp_gather_enabled,
|
||||
)
|
||||
from sglang.srt.model_executor.cuda_graph_config import Backend
|
||||
from sglang.srt.model_executor.forward_batch_deepseek_mha_mixin import (
|
||||
ForwardBatchDeepSeekMHAMixin,
|
||||
)
|
||||
@@ -59,6 +60,7 @@ from sglang.srt.runtime_context import (
|
||||
get_flags,
|
||||
get_lora,
|
||||
get_parallel,
|
||||
get_platform,
|
||||
mamba_cache_chunk_size,
|
||||
)
|
||||
from sglang.srt.speculative.spec_info import SpecInputType
|
||||
@@ -78,7 +80,7 @@ if TYPE_CHECKING:
|
||||
from sglang.srt.layers.engram_staging import StagedRows
|
||||
from sglang.srt.layers.layer_boundary.residual.stream import ResidualStream
|
||||
from sglang.srt.layers.logits_processor import LogitsProcessorOutput
|
||||
from sglang.srt.managers.schedule_batch import MultimodalInputs, ScheduleBatch
|
||||
from sglang.srt.managers.schedule_batch import MultimodalInputs, Req, ScheduleBatch
|
||||
from sglang.srt.mem_cache.kv_loc_plan import Cols, KVLocPlan
|
||||
from sglang.srt.model_executor.model_runner import ModelRunner
|
||||
from sglang.srt.sampling.sampling_batch_info import SamplingBatchInfo
|
||||
@@ -489,6 +491,17 @@ class NgramEmbeddingInfo:
|
||||
)
|
||||
|
||||
|
||||
def prompt_rest_lens_of(
|
||||
reqs: List[Req], seq_lens: List[int], decoding_req_ids: Set[int]
|
||||
) -> List[int]:
|
||||
"""Prompt tokens each request prefills after this extend, from the batch's own
|
||||
ends (an encoder replay batch ends at its prefix); a decode row has none."""
|
||||
return [
|
||||
0 if id(req) in decoding_req_ids else len(req.full_untruncated_fill_ids) - end
|
||||
for req, end in zip(reqs, seq_lens, strict=True)
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
"""Store all inputs of a forward pass."""
|
||||
@@ -669,6 +682,8 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
extend_prefix_lens_cpu: Optional[List[int]] = None
|
||||
extend_seq_lens_cpu: Optional[List[int]] = None
|
||||
extend_logprob_start_lens_cpu: Optional[List[int]] = None
|
||||
# Prompt tokens each request prefills after this extend (decoder SWA tail rows).
|
||||
extend_prompt_rest_lens_cpu: Optional[List[int]] = None
|
||||
extend_input_logprob_token_ids_gpu: Optional[torch.Tensor] = None
|
||||
|
||||
# For DP attention (MLP sync sizes)
|
||||
@@ -681,6 +696,9 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
# Has to be None when cuda graph is captured.
|
||||
global_num_tokens_for_logprob_cpu: Optional[List[int]] = None
|
||||
global_num_tokens_for_logprob_gpu: Optional[torch.Tensor] = None
|
||||
# Per DP rank, the rows the decoder SWA tail drops before the late layers;
|
||||
# None when no rank trims (see DeepseekV4Model's late-layer DP switch).
|
||||
global_decoder_trim_rows_cpu: Optional[List[int]] = None
|
||||
|
||||
# Real (non-padding) token count, held at two scopes whose meaning never
|
||||
# changes once set:
|
||||
@@ -1015,6 +1033,7 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
batch.seq_lens_sum = int(seq_lens_cpu.sum())
|
||||
|
||||
contains_last_prefill_chunk = False
|
||||
prompt_rest_lens = None
|
||||
if batch.forward_mode.is_extend_without_speculative():
|
||||
decoding_req_ids = {id(req) for req in (batch.decoding_reqs or ())}
|
||||
contains_last_prefill_chunk = any(
|
||||
@@ -1022,6 +1041,14 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
for req in batch.reqs
|
||||
if id(req) not in decoding_req_ids
|
||||
)
|
||||
if (
|
||||
envs.SGLANG_OPT_DSV41_PROMPT_WINDOW_TAIL.get()
|
||||
and get_parallel().attn_cp_size == 1
|
||||
and seq_lens_cpu is not None
|
||||
):
|
||||
prompt_rest_lens = prompt_rest_lens_of(
|
||||
batch.reqs, seq_lens_cpu.tolist(), decoding_req_ids
|
||||
)
|
||||
|
||||
from sglang.srt.model_executor.model_runner import ModelRunner
|
||||
|
||||
@@ -1082,6 +1109,7 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
global_forward_mode=batch.global_forward_mode,
|
||||
is_prefill_only=batch.is_prefill_only,
|
||||
contains_last_prefill_chunk=contains_last_prefill_chunk,
|
||||
extend_prompt_rest_lens_cpu=prompt_rest_lens,
|
||||
spec_algorithm=batch.spec_algorithm,
|
||||
capture_hidden_mode=capture_hidden_mode,
|
||||
return_hidden_states_before_norm=return_hidden_states_before_norm,
|
||||
@@ -1166,6 +1194,8 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
ret.init_mlp_sync_metadata(
|
||||
batch, device, is_draft_worker=model_runner.is_draft_worker
|
||||
)
|
||||
if not model_runner.is_draft_worker:
|
||||
ret.global_decoder_trim_rows_cpu = batch.global_decoder_trim_rows
|
||||
|
||||
if ret.forward_mode.is_idle():
|
||||
ret.positions = torch.empty((0,), dtype=torch.int64, device=device)
|
||||
@@ -1945,13 +1975,29 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
):
|
||||
local_late_rows = model_runner.attn_backend.late_layer_num_tokens(self)
|
||||
if local_late_rows is not None:
|
||||
late_rows = torch.empty(sync_group_size, dtype=torch.int64)
|
||||
all_gather_single(
|
||||
late_rows,
|
||||
torch.tensor([local_late_rows], dtype=torch.int64),
|
||||
group=get_parallel().tp_group.cpu_group,
|
||||
)
|
||||
self.global_late_layer_num_tokens_cpu = late_rows.tolist()
|
||||
trims = self.global_decoder_trim_rows_cpu
|
||||
if (
|
||||
trims is not None
|
||||
and get_platform().is_cuda
|
||||
and get_parallel().attn_tp_size == 1
|
||||
and get_exec().graph.cuda_graph_config.prefill.backend
|
||||
== Backend.DISABLED
|
||||
and not self.dp_padding_mode.is_max_len()
|
||||
):
|
||||
counts = [
|
||||
n - t for n, t in zip(global_num_tokens, trims, strict=True)
|
||||
]
|
||||
assert min(counts) >= 0, (global_num_tokens, trims)
|
||||
assert counts[dp_slot_in(counts)] == local_late_rows
|
||||
self.global_late_layer_num_tokens_cpu = counts
|
||||
else:
|
||||
late_rows = torch.empty(sync_group_size, dtype=torch.int64)
|
||||
all_gather_single(
|
||||
late_rows,
|
||||
torch.tensor([local_late_rows], dtype=torch.int64),
|
||||
group=get_parallel().tp_group.cpu_group,
|
||||
)
|
||||
self.global_late_layer_num_tokens_cpu = late_rows.tolist()
|
||||
|
||||
TboForwardBatchPreparer.prepare(
|
||||
batch=self, is_draft_worker=model_runner.is_draft_worker
|
||||
@@ -2074,6 +2120,10 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
|
||||
self.extend_logprob_start_lens_cpu.extend(
|
||||
[0] * (bs - len(self.extend_logprob_start_lens_cpu))
|
||||
)
|
||||
if self.extend_prompt_rest_lens_cpu is not None:
|
||||
self.extend_prompt_rest_lens_cpu.extend(
|
||||
[0] * (bs - len(self.extend_prompt_rest_lens_cpu))
|
||||
)
|
||||
|
||||
if self.rids_int is not None:
|
||||
self.rids_int = self._pad_tensor_to_size(self.rids_int, bs)
|
||||
|
||||
@@ -468,8 +468,6 @@ class ModelRunner:
|
||||
# Read-done mailbox: the scheduler's WAR barrier reads it from the runner
|
||||
# its worker names, and treats None as the coarse whole-forward fence.
|
||||
self.shared_read_done_event: Optional[torch.cuda.Event] = None
|
||||
# Set by the scheduler when every KV offload copy waits for forward_stream.
|
||||
self.kv_offload_fenced_on_forward_stream = False
|
||||
|
||||
# CPU offload
|
||||
set_offloader(create_offloader())
|
||||
@@ -1848,7 +1846,6 @@ class ModelRunner:
|
||||
else contextlib.nullcontext()
|
||||
)
|
||||
|
||||
_fwd_tail_t = time.monotonic_ns()
|
||||
with (
|
||||
canary_ctx,
|
||||
step_span_ctx,
|
||||
@@ -1910,12 +1907,6 @@ class ModelRunner:
|
||||
):
|
||||
self.maybe_join_ep_ranks()
|
||||
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline as _ft
|
||||
|
||||
_ft.detail("fwd_runner_tail", _fwd_tail_t, time.monotonic_ns())
|
||||
except Exception:
|
||||
pass
|
||||
return output
|
||||
|
||||
def _maybe_execute_deferred_mamba_cow_and_clear(
|
||||
@@ -2016,7 +2007,6 @@ class ModelRunner:
|
||||
)
|
||||
return ModelRunnerOutput(logits_output=ret, can_run_graph=can_run_graph)
|
||||
|
||||
_fwd_prep_t = time.monotonic_ns()
|
||||
# DP / MLP-sync padding + attn-tp normalization. Only the decode
|
||||
# cuda-graph path above pre-pads its static buffers and returns
|
||||
# early; split prefill, the prefill cuda graph, and the eager
|
||||
@@ -2033,21 +2023,6 @@ class ModelRunner:
|
||||
if dwdp_mgr is not None:
|
||||
dwdp_mgr.prefetch_first_layers()
|
||||
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline
|
||||
except Exception: # pragma: no cover
|
||||
fwd_timeline = None
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail(
|
||||
"fwd_prepare", _fwd_prep_t, time.monotonic_ns()
|
||||
)
|
||||
_fwd_exec_t = time.monotonic_ns()
|
||||
_fwd_ticket = (
|
||||
fwd_timeline.before_execute()
|
||||
if fwd_timeline is not None
|
||||
else None
|
||||
)
|
||||
|
||||
if forward_batch.forward_mode.is_split_prefill():
|
||||
# Layer-split mode; stays on ModelRunner, not the eager runner.
|
||||
ret = self.forward_split_prefill(
|
||||
@@ -2086,20 +2061,11 @@ class ModelRunner:
|
||||
forward_batch, pp_proxy_tensors=pp_proxy_tensors
|
||||
)
|
||||
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.after_execute(_fwd_ticket)
|
||||
fwd_timeline.detail(
|
||||
"fwd_execute", _fwd_exec_t, time.monotonic_ns()
|
||||
)
|
||||
_fwd_post_t = time.monotonic_ns()
|
||||
|
||||
if (
|
||||
forward_batch.global_num_tokens_cpu is not None
|
||||
and self.pp_group.is_last_rank
|
||||
):
|
||||
forward_batch.post_forward_mlp_sync_batch(ret)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail("fwd_post", _fwd_post_t, time.monotonic_ns())
|
||||
|
||||
return ModelRunnerOutput(logits_output=ret, can_run_graph=can_run_graph)
|
||||
|
||||
|
||||
@@ -38,7 +38,6 @@ Backend selection comes from cuda_graph_config.prefill:
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import time
|
||||
import dataclasses
|
||||
import inspect
|
||||
import logging
|
||||
@@ -2339,18 +2338,9 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
|
||||
) -> Optional[Union[LogitsProcessorOutput, PPProxyTensors, EmbeddingPoolerOutput]]:
|
||||
self._validate_capture_hidden_mode(forward_batch)
|
||||
with self.model_context(), self.backend.replay_session():
|
||||
try:
|
||||
from sglang.srt.observability import fwd_timeline
|
||||
|
||||
_gl_t = time.monotonic_ns()
|
||||
except Exception: # pragma: no cover
|
||||
fwd_timeline = None
|
||||
static_forward_batch, shared_read_ends = self.load_batch(
|
||||
forward_batch, **kwargs
|
||||
)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail("fwd_graph_load", _gl_t, time.monotonic_ns())
|
||||
_gr_t = time.monotonic_ns()
|
||||
static_num_tokens = len(static_forward_batch.input_ids)
|
||||
raw_num_tokens = self.raw_num_tokens
|
||||
shape_key = self._shape_key(static_num_tokens, forward_batch)
|
||||
@@ -2392,8 +2382,4 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
|
||||
raw_num_tokens,
|
||||
**kwargs,
|
||||
)
|
||||
if fwd_timeline is not None:
|
||||
fwd_timeline.detail(
|
||||
"fwd_graph_replay", _gr_t, time.monotonic_ns()
|
||||
)
|
||||
return self._finalize_execute_output(output)
|
||||
|
||||
@@ -31,12 +31,12 @@ def maybe_publish_prefill_shared_read_done(
|
||||
if not model_runner.spec_algorithm.supports_prefill_shared_read_done():
|
||||
return
|
||||
# Host and storage offload copy a stashed chunk's KV ordered only behind the
|
||||
# schedule stream, so they need the whole-forward barrier unless every such
|
||||
# copy waits for the forward stream itself.
|
||||
# schedule stream, so they need the whole-forward barrier.
|
||||
memory = get_memory()
|
||||
if memory.enable_lmcache or (
|
||||
(memory.enable_hierarchical_cache or memory.hicache_storage_backend is not None)
|
||||
and not model_runner.kv_offload_fenced_on_forward_stream
|
||||
if (
|
||||
memory.enable_hierarchical_cache
|
||||
or memory.hicache_storage_backend is not None
|
||||
or memory.enable_lmcache
|
||||
):
|
||||
return
|
||||
logger.info_once(
|
||||
|
||||
@@ -3,7 +3,6 @@ from __future__ import annotations
|
||||
import concurrent.futures
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from array import array
|
||||
from contextlib import contextmanager, nullcontext
|
||||
@@ -274,17 +273,6 @@ def get_mhc_ops() -> MhcOps:
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Optional vision-tower timing (env-gated; off by default).
|
||||
# SGLANG_MM_TRACE=1 emit one OTel span + one log line per get_image_feature call
|
||||
# SGLANG_MM_TRACE_SYNC=1 synchronize CUDA around ViT/aligner so the times are GPU time
|
||||
# (only for short vision-focused runs; leave 0 under load)
|
||||
# ---------------------------------------------------------------------------
|
||||
_MM_TRACE = os.getenv("SGLANG_MM_TRACE", "0").strip().lower() not in ("", "0", "false", "no", "off")
|
||||
_MM_TRACE_SYNC = os.getenv("SGLANG_MM_TRACE_SYNC", "0").strip().lower() not in ("", "0", "false", "no", "off")
|
||||
_MM_TRACE_RIDS = None
|
||||
_MM_TRACE_ACC = None
|
||||
|
||||
_FP8_WO_A_GEMM = envs.SGLANG_OPT_FP8_WO_A_GEMM.get()
|
||||
_FP8_WO_A_UE8M0 = _FP8_WO_A_GEMM and DEEPGEMM_SCALE_UE8M0
|
||||
|
||||
@@ -4766,62 +4754,6 @@ class DeepseekV4ForCausalLM(nn.Module):
|
||||
return pad_mm_tokens(input_ids, mm_inputs)
|
||||
|
||||
def get_image_feature(self, items):
|
||||
if not _MM_TRACE:
|
||||
return self._get_image_feature_impl(items)
|
||||
|
||||
global _MM_TRACE_ACC
|
||||
from opentelemetry import trace as _otel_trace
|
||||
|
||||
parallel = get_parallel()
|
||||
rids = list(_MM_TRACE_RIDS or [])
|
||||
acc = {
|
||||
"vit_s": 0.0,
|
||||
"aligner_s": 0.0,
|
||||
"items": 0,
|
||||
"patches": 0,
|
||||
"cuda": torch.cuda.is_available(),
|
||||
}
|
||||
prev_acc, _MM_TRACE_ACC = _MM_TRACE_ACC, acc
|
||||
tracer = _otel_trace.get_tracer("sglang.mm")
|
||||
try:
|
||||
if _MM_TRACE_SYNC and acc["cuda"]:
|
||||
torch.cuda.synchronize()
|
||||
t0 = time.perf_counter()
|
||||
with tracer.start_as_current_span("mm.vision_tower") as span:
|
||||
try:
|
||||
out = self._get_image_feature_impl(items)
|
||||
finally:
|
||||
if _MM_TRACE_SYNC and acc["cuda"]:
|
||||
torch.cuda.synchronize()
|
||||
total_s = time.perf_counter() - t0
|
||||
dp_rank = int(getattr(parallel, "attn_dp_rank", -1))
|
||||
span.set_attribute("sglang.dp_rank", dp_rank)
|
||||
span.set_attribute("sglang.n_items", int(acc["items"]))
|
||||
span.set_attribute("sglang.n_patches", int(acc["patches"]))
|
||||
span.set_attribute("sglang.vit_ms", round(acc["vit_s"] * 1e3, 3))
|
||||
span.set_attribute("sglang.aligner_ms", round(acc["aligner_s"] * 1e3, 3))
|
||||
span.set_attribute("sglang.total_ms", round(total_s * 1e3, 3))
|
||||
span.set_attribute("sglang.timing", "gpu_sync" if _MM_TRACE_SYNC else "cpu_wall_launch")
|
||||
span.set_attribute("sglang.n_rids", len(rids))
|
||||
if rids:
|
||||
span.set_attribute("sglang.rids", ",".join(map(str, rids[:8])))
|
||||
logger.info(
|
||||
"[MMTrace] dp=%d items=%d patches=%d vit_ms=%.3f aligner_ms=%.3f "
|
||||
"total_ms=%.3f timing=%s rids=%s",
|
||||
dp_rank,
|
||||
acc["items"],
|
||||
acc["patches"],
|
||||
acc["vit_s"] * 1e3,
|
||||
acc["aligner_s"] * 1e3,
|
||||
total_s * 1e3,
|
||||
"gpu_sync" if _MM_TRACE_SYNC else "cpu_wall_launch",
|
||||
",".join(map(str, rids[:4])),
|
||||
)
|
||||
return out
|
||||
finally:
|
||||
_MM_TRACE_ACC = prev_acc
|
||||
|
||||
def _get_image_feature_impl(self, items):
|
||||
"""Return complete spans for the shared MM cache and chunk scheduler."""
|
||||
|
||||
spans = []
|
||||
@@ -4836,21 +4768,7 @@ class DeepseekV4ForCausalLM(nn.Module):
|
||||
if plan is not None
|
||||
else pixels.to(dtype)
|
||||
)
|
||||
if _MM_TRACE_ACC is not None:
|
||||
t_vit = time.perf_counter()
|
||||
vision_features = self.vision(patches, h, w)
|
||||
if _MM_TRACE_SYNC and _MM_TRACE_ACC["cuda"]:
|
||||
torch.cuda.synchronize()
|
||||
_MM_TRACE_ACC["vit_s"] += time.perf_counter() - t_vit
|
||||
t_aligner = time.perf_counter()
|
||||
features = self.aligner(vision_features, h, w)
|
||||
if _MM_TRACE_SYNC and _MM_TRACE_ACC["cuda"]:
|
||||
torch.cuda.synchronize()
|
||||
_MM_TRACE_ACC["aligner_s"] += time.perf_counter() - t_aligner
|
||||
_MM_TRACE_ACC["items"] += 1
|
||||
_MM_TRACE_ACC["patches"] += int(h) * int(w)
|
||||
else:
|
||||
features = self.aligner(self.vision(patches, h, w), h, w)
|
||||
features = self.aligner(self.vision(patches, h, w), h, w)
|
||||
r = self.config.vision_downsample_ratio
|
||||
types = image_token_types((h + r - 1) // r, (w + r - 1) // r).to(device)
|
||||
span = torch.empty(
|
||||
@@ -4864,25 +4782,18 @@ class DeepseekV4ForCausalLM(nn.Module):
|
||||
return spans
|
||||
|
||||
def _prepare_mm_embeddings(self, input_ids, forward_batch):
|
||||
global _MM_TRACE_RIDS
|
||||
prev_rids = _MM_TRACE_RIDS
|
||||
if _MM_TRACE:
|
||||
_MM_TRACE_RIDS = list(getattr(forward_batch, "rids", None) or [])
|
||||
try:
|
||||
# Keep scheduler hash IDs intact: the shared embedder clamps its input in place.
|
||||
input_embeds, _ = embed_mm_inputs(
|
||||
mm_inputs_list=[
|
||||
item if item is not None else MultimodalInputs(mm_items=[])
|
||||
for item in forward_batch.mm_inputs
|
||||
],
|
||||
extend_prefix_lens=forward_batch.extend_prefix_lens_cpu,
|
||||
extend_seq_lens=forward_batch.extend_seq_lens_cpu,
|
||||
input_ids=input_ids.clone(),
|
||||
input_embedding=self.get_input_embeddings(),
|
||||
multimodal_model=self,
|
||||
)
|
||||
finally:
|
||||
_MM_TRACE_RIDS = prev_rids
|
||||
# Keep scheduler hash IDs intact: the shared embedder clamps its input in place.
|
||||
input_embeds, _ = embed_mm_inputs(
|
||||
mm_inputs_list=[
|
||||
item if item is not None else MultimodalInputs(mm_items=[])
|
||||
for item in forward_batch.mm_inputs
|
||||
],
|
||||
extend_prefix_lens=forward_batch.extend_prefix_lens_cpu,
|
||||
extend_seq_lens=forward_batch.extend_seq_lens_cpu,
|
||||
input_ids=input_ids.clone(),
|
||||
input_embedding=self.get_input_embeddings(),
|
||||
multimodal_model=self,
|
||||
)
|
||||
forward_batch.mm_input_embeds = input_embeds
|
||||
return input_embeds
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,164 +0,0 @@
|
||||
"""Opt-in GPU busy-window probe for prefill forwards (SGLANG_FWD_TIMELINE=1).
|
||||
|
||||
The scheduler/loop spans tell us *host* time; they cannot tell whether the GPU
|
||||
was busy while the host was working. This probe records one CUDA event pair
|
||||
around the model execute call and writes the resolved wall-clock window of the
|
||||
GPU work as one JSONL line per forward::
|
||||
|
||||
{"seq": 123, "rank": 0, "a_ns": ..., "b_ns": ..., "dur_ms": 42.1}
|
||||
|
||||
``a_ns``/``b_ns`` are ``time.time_ns()``-based wall clock nanoseconds, anchored
|
||||
to a reference event recorded once (after a ``cudaSynchronize``) so they can be
|
||||
joined with the loop/pass spans from OpenSearch. Everything before ``a_ns``
|
||||
and after ``b_ns`` in the same loop is GPU-idle (i.e. host-serial) time.
|
||||
|
||||
Design constraints:
|
||||
|
||||
* never raise into the scheduler -- any error disables the probe permanently;
|
||||
* the resolve step only reads events that are already complete
|
||||
(``Event.query()``), so it never blocks the scheduler thread;
|
||||
* file is per DP rank under ``/tmp`` (the container is hostPath-mounted for
|
||||
other reasons; ``kubectl exec ... cat`` is enough to collect it).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
|
||||
_ENABLED = os.environ.get("SGLANG_FWD_TIMELINE", "0") == "1"
|
||||
|
||||
try: # pragma: no cover - import guard only
|
||||
import torch
|
||||
except Exception: # pragma: no cover
|
||||
torch = None
|
||||
|
||||
_REF = None # reference CUDA event
|
||||
_REF_WALL_NS = 0
|
||||
_RANK = -1
|
||||
_PENDING = [] # [(seq, ev_a, ev_b)]
|
||||
_FILE = None
|
||||
_FAILED = False
|
||||
|
||||
|
||||
def enabled() -> bool:
|
||||
return _ENABLED and not _FAILED
|
||||
|
||||
|
||||
def _init() -> bool:
|
||||
"""Record the reference event; returns False when the probe must stay off."""
|
||||
global _REF, _REF_WALL_NS, _RANK, _FILE, _FAILED, _ENABLED
|
||||
if _REF is not None:
|
||||
return True
|
||||
if not _ENABLED or torch is None:
|
||||
return False
|
||||
try:
|
||||
from sglang.srt.distributed import get_parallel
|
||||
|
||||
_RANK = int(get_parallel().attn_dp_rank or 0)
|
||||
except Exception:
|
||||
_RANK = 0
|
||||
try:
|
||||
torch.cuda.synchronize()
|
||||
_REF_WALL_NS = time.time_ns()
|
||||
_REF = torch.cuda.Event(enable_timing=True)
|
||||
_REF.record()
|
||||
path = f"/tmp/fwd_timeline_rank{_RANK}.jsonl"
|
||||
_FILE = open(path, "a", buffering=1)
|
||||
_FILE.write(
|
||||
json.dumps({"header": True, "rank": _RANK, "ref_ns": _REF_WALL_NS}) + "\n"
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
_ENABLED = False
|
||||
_FAILED = True
|
||||
return False
|
||||
|
||||
|
||||
def before_execute():
|
||||
"""Record the GPU-start marker; returns an opaque ticket (or None)."""
|
||||
if not enabled():
|
||||
return None
|
||||
try:
|
||||
if not _init():
|
||||
return None
|
||||
ev = torch.cuda.Event(enable_timing=True)
|
||||
ev.record()
|
||||
return ev
|
||||
except Exception:
|
||||
_disable()
|
||||
return None
|
||||
|
||||
|
||||
def after_execute(ticket) -> None:
|
||||
if ticket is None or not enabled():
|
||||
return
|
||||
try:
|
||||
ev = torch.cuda.Event(enable_timing=True)
|
||||
ev.record()
|
||||
seq = 0
|
||||
try:
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
|
||||
seq = int(batch_telemetry.current_loop_seq() or 0)
|
||||
except Exception:
|
||||
pass
|
||||
_PENDING.append((seq, ticket, ev))
|
||||
except Exception:
|
||||
_disable()
|
||||
|
||||
|
||||
def detail(
|
||||
name: str, start_ns: int, end_ns: int, thread_cpu_ns=None
|
||||
) -> None:
|
||||
"""Attach a host phase window to the current loop span (absolute times)."""
|
||||
try:
|
||||
from sglang.srt.observability import batch_telemetry
|
||||
|
||||
batch_telemetry.detail_add(
|
||||
name, start_ns, end_ns, thread_cpu_ns=thread_cpu_ns
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def drain() -> None:
|
||||
"""Resolve completed windows (never blocks) and append them to the file."""
|
||||
if not _PENDING or _FILE is None:
|
||||
return
|
||||
try:
|
||||
keep = []
|
||||
lines = []
|
||||
for seq, ev_a, ev_b in _PENDING:
|
||||
try:
|
||||
ready = ev_b.query()
|
||||
except Exception:
|
||||
ready = False
|
||||
if not ready:
|
||||
keep.append((seq, ev_a, ev_b))
|
||||
continue
|
||||
a_ns = _REF_WALL_NS + int(_REF.elapsed_time(ev_a) * 1e6)
|
||||
b_ns = _REF_WALL_NS + int(_REF.elapsed_time(ev_b) * 1e6)
|
||||
lines.append(
|
||||
json.dumps(
|
||||
{
|
||||
"seq": seq,
|
||||
"rank": _RANK,
|
||||
"a_ns": a_ns,
|
||||
"b_ns": b_ns,
|
||||
"dur_ms": round((b_ns - a_ns) / 1e6, 3),
|
||||
}
|
||||
)
|
||||
)
|
||||
_PENDING[:] = keep
|
||||
if lines:
|
||||
_FILE.write("\n".join(lines) + "\n")
|
||||
except Exception:
|
||||
_disable()
|
||||
|
||||
|
||||
def _disable() -> None:
|
||||
global _ENABLED, _FAILED
|
||||
_ENABLED = False
|
||||
_FAILED = True
|
||||
@@ -562,34 +562,6 @@ class SchedulerMetricsCollector(_StatLoggerDIMixin):
|
||||
documentation="Total number of retracted output tokens.",
|
||||
labelnames=labels.keys(),
|
||||
)
|
||||
# Prefix-cache component gate (SWA window): Full-KV tokens that a
|
||||
# component validator refused to match, so they are re-prefilled.
|
||||
# Counted once per prefill admission (not on waiting-queue probes, PD
|
||||
# decode matches or HiCache rematches). With the SWA prefix-match gate
|
||||
# enabled this is the hit length lost to a missing sliding window; with
|
||||
# the gate disabled it only captures the host-only / load-back-gap
|
||||
# fallback, so it should be near 0.
|
||||
self.swa_gate_reject_tokens_total = Counter(
|
||||
name="sglang:swa_gate_reject_tokens_total",
|
||||
documentation=(
|
||||
"Full-KV matched tokens rejected by the SWA prefix-match gate "
|
||||
"at prefill admission and therefore re-prefilled."
|
||||
),
|
||||
labelnames=labels.keys(),
|
||||
)
|
||||
self.swa_gate_reject_admissions_total = Counter(
|
||||
name="sglang:swa_gate_reject_admissions_total",
|
||||
documentation=(
|
||||
"Prefill admissions whose Full-KV match was truncated by the "
|
||||
"SWA prefix-match gate. A retracted and re-admitted request is "
|
||||
"counted again because it is re-prefilled."
|
||||
),
|
||||
labelnames=labels.keys(),
|
||||
)
|
||||
# Create the zero-valued children so a gate-off deployment exposes the
|
||||
# series before the first rejection.
|
||||
self.swa_gate_reject_tokens_total.labels(**self.labels).inc(0)
|
||||
self.swa_gate_reject_admissions_total.labels(**self.labels).inc(0)
|
||||
self.num_paused_reqs = Gauge(
|
||||
name="sglang:num_paused_reqs",
|
||||
documentation="The number of paused requests by async weight sync.",
|
||||
@@ -1450,12 +1422,6 @@ class SchedulerMetricsCollector(_StatLoggerDIMixin):
|
||||
if count > 0:
|
||||
self.num_prefill_retries_total.labels(**self.labels).inc(count)
|
||||
|
||||
def observe_swa_gate_reject_admission(self, tokens: int) -> None:
|
||||
"""Record one prefill admission whose match the SWA gate truncated."""
|
||||
if tokens > 0:
|
||||
self.swa_gate_reject_tokens_total.labels(**self.labels).inc(tokens)
|
||||
self.swa_gate_reject_admissions_total.labels(**self.labels).inc(1)
|
||||
|
||||
def observe_kv_transfer_metrics(
|
||||
self,
|
||||
latency_ms: float,
|
||||
|
||||
@@ -5,7 +5,6 @@ parses them, and tool-call arguments arrive as the API streams them."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from typing import Any, Optional
|
||||
|
||||
@@ -18,11 +17,6 @@ from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
load_deepseek_recipe,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# The most tokens one UTF-8 character can span, a byte each.
|
||||
_MAX_UTF8_BYTES = 4
|
||||
|
||||
|
||||
class RecipeToolCallDelta(msgspec.Struct, frozen=True):
|
||||
"""A piece of one call's JSON arguments; the first piece also names the call."""
|
||||
@@ -80,7 +74,6 @@ class DeepSeekRecipeOutputParser:
|
||||
separate_reasoning: bool = True,
|
||||
stream_reasoning: bool = True,
|
||||
stop: Sequence[str] = (),
|
||||
callable_names: Optional[frozenset[str]] = None,
|
||||
):
|
||||
recipe = load_deepseek_recipe()
|
||||
options = recipe.ParsingOptions(
|
||||
@@ -108,10 +101,6 @@ class DeepSeekRecipeOutputParser:
|
||||
# out, so a matched stop is cut before recipe has emitted any of it.
|
||||
self._stop_hold_chars = max(map(len, stop), default=1) - 1
|
||||
self._pending_ids: list[int] = []
|
||||
self._callable_names = callable_names
|
||||
# recipe's index of each call so far, mapped to the index it is reported
|
||||
# at; None for a dropped call.
|
||||
self._call_indices: dict[int, Optional[int]] = {}
|
||||
self.num_consumed_tokens = 0
|
||||
|
||||
def feed(self, token_ids: Sequence[int]) -> RecipeOutputDelta:
|
||||
@@ -203,27 +192,6 @@ class DeepSeekRecipeOutputParser:
|
||||
+ self._hf_tokenizer.encode(text[:cut], add_special_tokens=False).ids
|
||||
)
|
||||
|
||||
def _reported_call(
|
||||
self, call: RecipeToolCallDelta
|
||||
) -> Optional[RecipeToolCallDelta]:
|
||||
"""`call` at the index it is reported at; None when it belongs to a call
|
||||
outside `callable_names`."""
|
||||
if self._callable_names is None:
|
||||
return call
|
||||
if call.name is not None:
|
||||
index = None
|
||||
if call.name in self._callable_names:
|
||||
index = sum(kept is not None for kept in self._call_indices.values())
|
||||
else:
|
||||
logger.warning(
|
||||
"Model attempted to call undefined function: %s", call.name
|
||||
)
|
||||
self._call_indices[call.index] = index
|
||||
index = self._call_indices[call.index]
|
||||
if index is None:
|
||||
return None
|
||||
return msgspec.structs.replace(call, index=index)
|
||||
|
||||
def _deltas(self, chunks: list[Any]) -> RecipeOutputDelta:
|
||||
delta = RecipeOutputDelta()
|
||||
for chunk in chunks:
|
||||
@@ -234,19 +202,12 @@ class DeepSeekRecipeOutputParser:
|
||||
reasoning = piece.get("reasoning_content") or ""
|
||||
content = piece.get("content") or ""
|
||||
tool_calls = tuple(
|
||||
call
|
||||
for call in map(
|
||||
self._reported_call,
|
||||
(
|
||||
RecipeToolCallDelta(
|
||||
index=call["index"],
|
||||
arguments=call["function"]["arguments"],
|
||||
name=call["function"].get("name"),
|
||||
)
|
||||
for call in piece.get("tool_calls") or ()
|
||||
),
|
||||
RecipeToolCallDelta(
|
||||
index=call["index"],
|
||||
arguments=call["function"]["arguments"],
|
||||
name=call["function"].get("name"),
|
||||
)
|
||||
if call is not None
|
||||
for call in piece.get("tool_calls") or ()
|
||||
)
|
||||
if self._reasoning_as_content:
|
||||
content, reasoning = reasoning + content, ""
|
||||
|
||||
@@ -6,7 +6,6 @@ from __future__ import annotations
|
||||
import copy
|
||||
import functools
|
||||
import json
|
||||
import logging
|
||||
from types import ModuleType
|
||||
from typing import Any, Optional, Union
|
||||
|
||||
@@ -16,8 +15,6 @@ DEEPSEEK_RECIPE_INSTALL_HINT = (
|
||||
"`pip install deepseek-recipe`."
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODEL_NAMES = {"dsv4": "DeepSeek-V4-Pro", "dsv41": "DeepSeek-V4.1-Flash"}
|
||||
# The checkpoint tokenizer's image token; recipe's bundled tokenizer spells it <|image|>.
|
||||
IMAGE_TOKEN = "<|deepseek_image|>"
|
||||
@@ -98,23 +95,6 @@ def pre_normalize(body: dict[str, Any]) -> dict[str, Any]:
|
||||
return request
|
||||
|
||||
|
||||
def loaded_tools(conversation: Any) -> list[Any]:
|
||||
"""The tools the model can call after `conversation`'s history: those it does
|
||||
not defer and the deferred ones a system message has loaded."""
|
||||
recipe = load_deepseek_recipe()
|
||||
loaded = {
|
||||
tool.name
|
||||
for message in conversation.messages
|
||||
if isinstance(message, recipe.SystemMessage)
|
||||
for tool in message.tools or []
|
||||
}
|
||||
return [
|
||||
tool
|
||||
for tool in conversation.tools
|
||||
if not tool.defer_loading or tool.name in loaded
|
||||
]
|
||||
|
||||
|
||||
def _error_message(body: Any) -> str:
|
||||
try:
|
||||
return json.loads(body)["error"]["message"]
|
||||
@@ -130,20 +110,6 @@ class DeepSeekRecipe:
|
||||
recipe = load_deepseek_recipe()
|
||||
self.model = MODEL_NAMES[chat_encoding_spec]
|
||||
self.hf_tokenizer = tokenizer
|
||||
self._fast_prompt_tokenizer = None
|
||||
if type(tokenizer).__module__.partition(".")[0] == "fastokens":
|
||||
from fastokens import Tokenizer
|
||||
|
||||
if hasattr(Tokenizer, "encode_segments"):
|
||||
self._fast_prompt_tokenizer = Tokenizer.from_json_str(
|
||||
tokenizer.to_str()
|
||||
)
|
||||
logger.info("DeepSeek recipe prompt encoder: fastokens segmented")
|
||||
else:
|
||||
logger.warning(
|
||||
"deepseek-recipe prompt acceleration requires fastokens.encode_segments; "
|
||||
"using the reference prompt tokenizer"
|
||||
)
|
||||
self.special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
|
||||
self._options = recipe.ConversionOptions().with_special_tokens(
|
||||
self.special_tokens
|
||||
@@ -180,19 +146,6 @@ class DeepSeekRecipe:
|
||||
) -> list[int]:
|
||||
"""Prompt ids for `body` (see `pre_normalize`); a request recipe rejects
|
||||
raises ValueError. `image_urls` receives the images in placeholder order."""
|
||||
conversation = self.convert(body, effort_budget=effort_budget)
|
||||
prompt_ids = self.encode_conversation(conversation, effort_budget=effort_budget)
|
||||
if image_urls is not None:
|
||||
image_urls.extend(self.image_urls(conversation))
|
||||
if task is not None:
|
||||
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
|
||||
return prompt_ids
|
||||
|
||||
def convert(
|
||||
self, body: dict[str, Any], *, effort_budget: Optional[int] = None
|
||||
) -> Any:
|
||||
"""Convert `body` to recipe's `Conversation`; invalid requests raise ValueError.
|
||||
Pass the same `effort_budget` to `encode_conversation`."""
|
||||
recipe = load_deepseek_recipe()
|
||||
normalized = pre_normalize(body)
|
||||
normalized["model"] = self.model
|
||||
@@ -202,104 +155,16 @@ class DeepSeekRecipe:
|
||||
converted = recipe.ChatCompletionRequest(normalized).convert(self._options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
return converted.conversation
|
||||
|
||||
def convert_native(
|
||||
self, protocol: str, body: dict[str, Any], *, thinking: bool
|
||||
) -> Any:
|
||||
"""Convert a Responses or Messages request to recipe's `Conversation` in the
|
||||
host's thinking mode; invalid requests raise ValueError."""
|
||||
recipe = load_deepseek_recipe()
|
||||
request_type = {
|
||||
"responses": recipe.ResponsesRequest,
|
||||
"messages": recipe.MessagesRequest,
|
||||
}[protocol]
|
||||
options = self._options.with_default_thinking_mode(thinking)
|
||||
try:
|
||||
converted = request_type({**body, "model": self.model}).convert(options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
return converted.conversation
|
||||
|
||||
def with_host_settings(
|
||||
self,
|
||||
conversation: Any,
|
||||
*,
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> Any:
|
||||
"""`conversation` with what recipe's Responses and Messages conversions
|
||||
leave out: a JSON response schema, and SGLang's reading of a trailing
|
||||
assistant turn, continued or else re-sent as a user turn."""
|
||||
recipe = load_deepseek_recipe()
|
||||
messages = conversation.messages
|
||||
last = messages[-1] if messages else None
|
||||
if last is not None and last.role == "assistant":
|
||||
# recipe closes the turn and opens another; SGLang continues it or,
|
||||
# as `_handle_last_assistant_message`, re-sends it as a user turn.
|
||||
if continue_final_message:
|
||||
last = recipe.AssistantMessage(
|
||||
last.content, last.reasoning_content, prefix=True
|
||||
)
|
||||
else:
|
||||
last = recipe.UserMessage(last.content)
|
||||
messages = [*messages[:-1], last]
|
||||
elif response_schema is None:
|
||||
return conversation
|
||||
return recipe.Conversation(
|
||||
messages,
|
||||
thinking_mode=conversation.thinking_mode,
|
||||
tools=conversation.tools,
|
||||
tool_choice=conversation.tool_choice,
|
||||
reasoning_effort=conversation.reasoning_effort,
|
||||
response_schema=(
|
||||
response_schema
|
||||
if response_schema is not None
|
||||
else conversation.response_schema
|
||||
),
|
||||
)
|
||||
|
||||
def namespace_tool_names(
|
||||
self, namespaces: list[dict[str, Any]]
|
||||
) -> dict[str, tuple[str, str]]:
|
||||
"""The name recipe's Responses conversion calls each member of `namespaces`
|
||||
by, mapped to its namespace and its name there. `namespaces` lists the
|
||||
request's namespace tools in conversion order: declared, then discovered."""
|
||||
recipe = load_deepseek_recipe()
|
||||
body = {"model": self.model, "input": "", "tools": namespaces}
|
||||
try:
|
||||
return recipe.ResponsesRequest(body).namespace_tool_map(self._options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
|
||||
def image_urls(self, conversation: Any) -> list[str]:
|
||||
"""The images of `conversation` in the order its prompt places them."""
|
||||
rendered = self._encoding.render_conversation(conversation)
|
||||
return [source.url or source.data_url for source in rendered.image_sources]
|
||||
|
||||
def encode_conversation(
|
||||
self, conversation: Any, *, effort_budget: Optional[int] = None
|
||||
) -> list[int]:
|
||||
"""Encode a recipe `Conversation`, including a prefix of its messages."""
|
||||
if self._fast_prompt_tokenizer is None:
|
||||
prompt_ids = self._encoding.encode(conversation)
|
||||
else:
|
||||
rendered = self._encoding.render_conversation(conversation)
|
||||
prompt = rendered.prompt
|
||||
segments = []
|
||||
cursor = 0
|
||||
# Recipe distinguishes control tokens from identical spellings in
|
||||
# user text. Its Python ranges use code-point offsets, not bytes.
|
||||
for start, end in rendered.special_ranges:
|
||||
if start > cursor:
|
||||
segments.append((prompt[cursor:start], False))
|
||||
segments.append((prompt[start:end], True))
|
||||
cursor = end
|
||||
if cursor < len(prompt):
|
||||
segments.append((prompt[cursor:], False))
|
||||
prompt_ids = self._fast_prompt_tokenizer.encode_segments(segments).ids
|
||||
if effort_budget is not None and conversation.thinking_mode:
|
||||
prompt_ids = self._encoding.encode(converted.conversation)
|
||||
if image_urls is not None:
|
||||
rendered = self._encoding.render_conversation(converted.conversation)
|
||||
image_urls.extend(
|
||||
source.url or source.data_url for source in rendered.image_sources
|
||||
)
|
||||
if effort_budget is not None and body["thinking"]:
|
||||
prompt_ids = self._replace_effort_budget(prompt_ids, effort_budget)
|
||||
if task is not None:
|
||||
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
|
||||
return prompt_ids
|
||||
|
||||
def parse_start(
|
||||
|
||||
@@ -97,9 +97,7 @@ class AnthropicMessagesMixin:
|
||||
)
|
||||
|
||||
serving = AnthropicServing(
|
||||
openai_serving_chat=SimpleNamespace(
|
||||
supports_inline_system=True, native_tool_loading=False
|
||||
)
|
||||
openai_serving_chat=SimpleNamespace(supports_inline_system=True)
|
||||
)
|
||||
chat_request = serving._convert_to_chat_completion_request(anthropic_request)
|
||||
converted = chat_request.model_dump()
|
||||
|
||||
@@ -1,253 +0,0 @@
|
||||
"""CUDA ordering/timeline tests for the per-op async backup index copy.
|
||||
|
||||
Plan: handoff/prefill-backup-overlap-next-20261107.md §3/§8.2.
|
||||
|
||||
Run inside the GPU container with the overlay source:
|
||||
|
||||
PYTHONPATH=/tmp/ovr python -m pytest \
|
||||
test/registered/cuda/test_async_backup_index_copy_cuda.py -q
|
||||
|
||||
What is proven on real hardware:
|
||||
|
||||
1. ``_start_async_index_copy`` returns while a long kernel is still queued on
|
||||
the scheduler stream, and the synchronous ``.cpu()`` on that stream blocks;
|
||||
2. the copy is only released after that fence, but the *next* forward (a
|
||||
marker kernel on another stream, ordered behind the same fence) runs
|
||||
concurrently with it: the copy end timestamp falls strictly inside the
|
||||
marker's [start, end] interval;
|
||||
3. the host can enqueue the next batch before ``ready_event`` completes and
|
||||
the event is not reported ready early;
|
||||
4. with three ops whose copies complete out of order, the worker still merges
|
||||
them in op order, identical to ``CacheOperation.merge_ops`` over the CPU
|
||||
reference tensors;
|
||||
5. the copy stream is distinct from the scheduler stream.
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.srt.managers import cache_controller as cc_mod
|
||||
|
||||
|
||||
class _FakeHostPool:
|
||||
"""Not a HostKVCache: uses_shared_host_layout() is false, i.e. the sync path
|
||||
would call device_indices.cpu()."""
|
||||
|
||||
shared_allocation_domain = None
|
||||
|
||||
|
||||
class _FakeEntry:
|
||||
def __init__(self):
|
||||
self.host_pool = _FakeHostPool()
|
||||
|
||||
|
||||
class _FakeMemPoolHost:
|
||||
layout = "page_first_direct"
|
||||
|
||||
def __init__(self):
|
||||
self.anchor_entry = _FakeEntry()
|
||||
self.entry_map = {}
|
||||
|
||||
|
||||
def _make_controller():
|
||||
c = cc_mod.HiCacheController.__new__(cc_mod.HiCacheController)
|
||||
c.io_backend = "direct"
|
||||
c.mem_pool_host = _FakeMemPoolHost()
|
||||
c.write_queue = []
|
||||
c.load_queue = []
|
||||
c.ack_write_queue = []
|
||||
c.ack_load_queue = []
|
||||
c._indices_stream = None
|
||||
c._write_submit_queue = None
|
||||
c._write_result_queue = cc_mod.Queue()
|
||||
c._write_submit_stop = threading.Event()
|
||||
c._write_submit_thread = None
|
||||
c._write_submit_state_lock = threading.Lock()
|
||||
c._write_submit_cond = threading.Condition(c._write_submit_state_lock)
|
||||
c._write_submit_active = 0
|
||||
c._async_pinned_bytes = 0
|
||||
c._retire_parts = []
|
||||
return c
|
||||
|
||||
|
||||
def _op(device_indices, node_id):
|
||||
host = torch.empty(device_indices.numel(), dtype=torch.int64)
|
||||
return cc_mod.CacheOperation(host, device_indices, node_id)
|
||||
|
||||
|
||||
class AsyncBackupIndexCopyCudaTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cc_mod._ASYNC_BACKUP_SUBMIT = True
|
||||
cls.saved_pin = cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX
|
||||
cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX = 1 << 30
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX = cls.saved_pin
|
||||
|
||||
def test_start_does_not_block_and_copy_is_async(self):
|
||||
torch.cuda.synchronize()
|
||||
sched = torch.cuda.current_stream()
|
||||
fwd = torch.cuda.Stream()
|
||||
indices = torch.arange(4096, dtype=torch.int64, device="cuda")
|
||||
|
||||
# Simulated in-flight forward plus the scheduler's WAR fence.
|
||||
with torch.cuda.stream(fwd):
|
||||
torch.cuda._sleep(2_000_000_000)
|
||||
sched.wait_stream(fwd)
|
||||
|
||||
c = _make_controller()
|
||||
op = _op(indices, 11)
|
||||
t0 = time.perf_counter()
|
||||
c._start_async_index_copy(op)
|
||||
start_elapsed = time.perf_counter() - t0
|
||||
self.assertIsNotNone(op.async_parts)
|
||||
# Starting the copy must not block on the simulated forward.
|
||||
self.assertLess(start_elapsed, 0.05, start_elapsed)
|
||||
self.assertFalse(op.async_parts.ready_event.query())
|
||||
|
||||
# The synchronous baseline on the same stream does block.
|
||||
t1 = time.perf_counter()
|
||||
reference = indices.cpu()
|
||||
sync_elapsed = time.perf_counter() - t1
|
||||
self.assertGreater(sync_elapsed, 0.1, sync_elapsed)
|
||||
|
||||
op.async_parts.ready_event.synchronize()
|
||||
self.assertTrue(op.async_parts.ready_event.query())
|
||||
c._materialize_async_indices([op])
|
||||
self.assertTrue(bool(torch.equal(op.device_indices, reference)))
|
||||
self.assertFalse(op.device_indices.is_cuda)
|
||||
c._release_async_parts([op])
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
|
||||
def test_next_forward_overlaps_the_index_copy(self):
|
||||
torch.cuda.synchronize()
|
||||
sched = torch.cuda.current_stream()
|
||||
fwd = torch.cuda.Stream()
|
||||
# Large enough that the copy takes long enough to observe (32 MB).
|
||||
indices = torch.arange(4 * 1024 * 1024, dtype=torch.int64, device="cuda")
|
||||
|
||||
# 1. A long fence on the scheduler stream (the in-flight forward).
|
||||
with torch.cuda.stream(sched):
|
||||
fence_done = torch.cuda.Event(enable_timing=True)
|
||||
fence_done.record()
|
||||
torch.cuda._sleep(1_000_000_000)
|
||||
|
||||
# 2. Per-op pinned copy started now; it inherits the fence.
|
||||
c = _make_controller()
|
||||
op = _op(indices, 3)
|
||||
c._start_async_index_copy(op)
|
||||
self.assertIsNotNone(op.async_parts)
|
||||
|
||||
# 3. The next batch launch waits for the scheduler stream (as the real
|
||||
# forward_stream does) and then runs a long marker kernel.
|
||||
copy_end = torch.cuda.Event(enable_timing=True)
|
||||
with torch.cuda.stream(c._indices_stream):
|
||||
copy_end.record()
|
||||
launch_host_ns = time.monotonic_ns()
|
||||
with torch.cuda.stream(fwd):
|
||||
fwd.wait_stream(sched)
|
||||
fwd_start = torch.cuda.Event(enable_timing=True)
|
||||
fwd_start.record()
|
||||
torch.cuda._sleep(1_500_000_000)
|
||||
fwd_end = torch.cuda.Event(enable_timing=True)
|
||||
fwd_end.record()
|
||||
|
||||
# 4. The event must not be ready while the fence is in flight, and the
|
||||
# host has already enqueued the next batch.
|
||||
ready_seen_ns = [None]
|
||||
|
||||
def _wait_ready():
|
||||
op.async_parts.ready_event.synchronize()
|
||||
ready_seen_ns[0] = time.monotonic_ns()
|
||||
|
||||
waiter = threading.Thread(target=_wait_ready)
|
||||
waiter.start()
|
||||
self.assertIsNone(ready_seen_ns[0])
|
||||
waiter.join(timeout=30)
|
||||
self.assertIsNotNone(ready_seen_ns[0])
|
||||
self.assertLess(launch_host_ns, ready_seen_ns[0])
|
||||
|
||||
torch.cuda.synchronize()
|
||||
# CUDA timestamps: copy end strictly inside the next forward.
|
||||
self.assertGreater(fwd_start.elapsed_time(copy_end), 0.0)
|
||||
self.assertGreater(copy_end.elapsed_time(fwd_end), 0.0)
|
||||
self.assertGreater(fwd_start.elapsed_time(fwd_end), 0.5)
|
||||
|
||||
c._materialize_async_indices([op])
|
||||
self.assertTrue(
|
||||
bool(torch.equal(op.device_indices, indices.cpu()))
|
||||
)
|
||||
c._release_async_parts([op])
|
||||
|
||||
def test_copy_stream_is_distinct_from_scheduler(self):
|
||||
torch.cuda.synchronize()
|
||||
c = _make_controller()
|
||||
op = _op(torch.arange(8, dtype=torch.int64, device="cuda"), 0)
|
||||
c._start_async_index_copy(op)
|
||||
self.assertIsNotNone(c._indices_stream)
|
||||
self.assertNotEqual(
|
||||
c._indices_stream.cuda_stream,
|
||||
torch.cuda.current_stream().cuda_stream,
|
||||
)
|
||||
op.async_parts.ready_event.synchronize()
|
||||
c._release_async_parts([op])
|
||||
|
||||
def test_out_of_order_completion_keeps_op_order(self):
|
||||
torch.cuda.synchronize()
|
||||
c = _make_controller()
|
||||
delays = {0: 600_000_000, 1: 150_000_000, 2: 300_000_000}
|
||||
ops = []
|
||||
references = []
|
||||
for i in range(3):
|
||||
src = torch.arange(8, dtype=torch.int64, device="cuda") + i * 100
|
||||
op = _op(src, node_id=i)
|
||||
buf = torch.empty(src.shape, dtype=src.dtype, device="cpu", pin_memory=True)
|
||||
stream = torch.cuda.Stream()
|
||||
with torch.cuda.stream(stream):
|
||||
torch.cuda._sleep(delays[i])
|
||||
buf.copy_(src, non_blocking=True)
|
||||
ev = torch.cuda.Event()
|
||||
ev.record(stream)
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
[cc_mod.AsyncIndexCopy((None, None), src, buf, buf.numel() * 8)],
|
||||
ev,
|
||||
buf.numel() * 8,
|
||||
)
|
||||
ops.append(op)
|
||||
references.append(src.cpu())
|
||||
|
||||
captured = []
|
||||
c._submit_write_operation = lambda merged: (
|
||||
captured.append(merged.device_indices.clone())
|
||||
or cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
|
||||
)
|
||||
c._write_submit_queue = cc_mod.Queue(maxsize=4)
|
||||
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
|
||||
c._write_submit_thread = worker
|
||||
worker.start()
|
||||
try:
|
||||
c._write_submit_active = 1
|
||||
c._write_submit_queue.put(cc_mod.WriteSubmission(ops, None, 0))
|
||||
deadline = time.time() + 30
|
||||
while c._write_submit_active and time.time() < deadline:
|
||||
time.sleep(0.01)
|
||||
self.assertEqual(c._write_submit_active, 0)
|
||||
finally:
|
||||
c._write_submit_queue.put(None)
|
||||
worker.join(timeout=10)
|
||||
|
||||
self.assertEqual(len(captured), 1)
|
||||
expected = torch.cat(references)
|
||||
self.assertTrue(bool(torch.equal(captured[0], expected)))
|
||||
self.assertEqual(ops[0].node_ids, [0])
|
||||
self.assertEqual(ops[1].node_ids, [1])
|
||||
self.assertEqual(ops[2].node_ids, [2])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,550 +0,0 @@
|
||||
"""Unit tests for the async HiCache backup submission state machine.
|
||||
|
||||
Design: handoff/prefill-backup-overlap-next-20261107.md §8.1.
|
||||
|
||||
These tests exercise the scheduler/worker split without a GPU: the index copy
|
||||
itself is covered by the CUDA test, and here the device stream/event and the
|
||||
submit path are faked. What is checked:
|
||||
|
||||
* disabled / inapplicable shapes keep the sync path;
|
||||
* per-op pinned parts are materialized per (pool, indices_from_pool) key;
|
||||
* the worker waits every ready event before touching host buffers;
|
||||
* exactly-once ACK, fail-closed errors, bounded queues, idle/stop lifecycle;
|
||||
* has_inflight_device_transfers reflects queued/running results.
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from sglang.srt.managers import cache_controller as cc_mod
|
||||
from sglang.srt.mem_cache.hicache_storage import PoolName, PoolTransfer
|
||||
|
||||
|
||||
class _FakeEvent:
|
||||
def __init__(self, worker_log=None, name="ev"):
|
||||
self.worker_log = worker_log
|
||||
self.name = name
|
||||
self._set = False
|
||||
self.waited = 0
|
||||
|
||||
def record(self, stream=None):
|
||||
self._set = True
|
||||
|
||||
def synchronize(self):
|
||||
self.waited += 1
|
||||
if self.worker_log is not None:
|
||||
self.worker_log.append(("wait", self.name))
|
||||
self._set = True
|
||||
|
||||
def query(self):
|
||||
return self._set
|
||||
|
||||
|
||||
class _FakeStream:
|
||||
def __init__(self, name="stream", log=None):
|
||||
self.name = name
|
||||
self.log = log
|
||||
self.waited = 0
|
||||
|
||||
def wait_stream(self, other):
|
||||
self.waited += 1
|
||||
if self.log is not None:
|
||||
self.log.append(("wait_stream", self.name, getattr(other, "name", "?")))
|
||||
|
||||
def wait_event(self, event):
|
||||
pass
|
||||
|
||||
|
||||
class _FakeStreamCtx:
|
||||
def __init__(self, stream):
|
||||
self.stream = stream
|
||||
|
||||
def __enter__(self):
|
||||
return self.stream
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
class _FakeCudaModule:
|
||||
def __init__(self, log=None):
|
||||
self.log = log
|
||||
self.current = _FakeStream("schedule", log)
|
||||
|
||||
def Stream(self, *a, **k):
|
||||
return _FakeStream("indices", self.log)
|
||||
|
||||
def StreamContext(self, stream):
|
||||
return _FakeStreamCtx(stream)
|
||||
|
||||
def Event(self, *a, **k):
|
||||
return _FakeEvent(self.log)
|
||||
|
||||
def current_stream(self):
|
||||
return self.current
|
||||
|
||||
|
||||
class _FakeTensor:
|
||||
def __init__(self, numel=8, element_size=8, is_cuda=True, tag="dev"):
|
||||
self._numel = numel
|
||||
self._element_size = element_size
|
||||
self.is_cuda = is_cuda
|
||||
self.tag = tag
|
||||
self.recorded = []
|
||||
self.cpu_calls = 0
|
||||
self.device = types.SimpleNamespace(type="cuda" if is_cuda else "cpu")
|
||||
self.dtype = "torch.int64"
|
||||
|
||||
def numel(self):
|
||||
return self._numel
|
||||
|
||||
def element_size(self):
|
||||
return self._element_size
|
||||
|
||||
def record_stream(self, stream):
|
||||
self.recorded.append(stream)
|
||||
|
||||
def copy_(self, src, non_blocking=False):
|
||||
self.copied_from = src
|
||||
|
||||
@property
|
||||
def shape(self):
|
||||
return (self._numel,)
|
||||
|
||||
def cpu(self):
|
||||
self.cpu_calls += 1
|
||||
return _FakeTensor(self._numel, self._element_size, False, tag=self.tag + "-cpu")
|
||||
|
||||
def __len__(self):
|
||||
return self._numel
|
||||
|
||||
|
||||
class _FakeHostPool:
|
||||
def __init__(self, shared=False):
|
||||
self.shared = shared
|
||||
|
||||
|
||||
class _FakeEntry:
|
||||
def __init__(self, host_pool):
|
||||
self.host_pool = host_pool
|
||||
|
||||
|
||||
class _FakeMemPoolHost:
|
||||
layout = "page_first_direct"
|
||||
|
||||
def __init__(self, anchor_shared=False, sidecars=None):
|
||||
self.anchor_entry = _FakeEntry(_FakeHostPool(anchor_shared))
|
||||
self.entry_map = {
|
||||
name: _FakeEntry(pool) for name, pool in (sidecars or {}).items()
|
||||
}
|
||||
|
||||
|
||||
def _make_controller(*, enabled=True, anchor_shared=False, sidecars=None, policy=""):
|
||||
c = cc_mod.HiCacheController.__new__(cc_mod.HiCacheController)
|
||||
c.io_backend = "direct"
|
||||
c.write_policy = policy
|
||||
c.mem_pool_host = _FakeMemPoolHost(anchor_shared, sidecars)
|
||||
c.write_queue = []
|
||||
c.load_queue = []
|
||||
c.ack_write_queue = []
|
||||
c.ack_load_queue = []
|
||||
c._indices_stream = None
|
||||
c._write_submit_queue = None
|
||||
c._write_result_queue = cc_mod.Queue()
|
||||
c._write_submit_stop = threading.Event()
|
||||
c._write_submit_thread = None
|
||||
c._write_submit_state_lock = threading.Lock()
|
||||
c._write_submit_cond = threading.Condition(c._write_submit_state_lock)
|
||||
c._write_submit_active = 0
|
||||
c._async_pinned_bytes = 0
|
||||
c._retire_parts = []
|
||||
return c
|
||||
|
||||
|
||||
def _op(controller, *, anchor=None, transfers=(), node_id=7):
|
||||
host = _FakeTensor(8, 8, is_cuda=False, tag="host")
|
||||
dev = anchor or _FakeTensor(8, 8, is_cuda=True, tag="anchor")
|
||||
op = cc_mod.CacheOperation(host, dev, node_id, pool_transfers=list(transfers))
|
||||
return op
|
||||
|
||||
|
||||
class AsyncBackupUnitTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
patcher = mock.patch.object(cc_mod, "_ASYNC_BACKUP_SUBMIT", True)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
# The real predicate isinstance-checks HostKVCache; the fakes expose a
|
||||
# plain `shared` flag instead.
|
||||
shared_patcher = mock.patch.object(
|
||||
cc_mod, "uses_shared_host_layout", lambda pool: bool(getattr(pool, "shared", False))
|
||||
)
|
||||
shared_patcher.start()
|
||||
self.addCleanup(shared_patcher.stop)
|
||||
cat_patcher = mock.patch.object(
|
||||
cc_mod.torch,
|
||||
"cat",
|
||||
side_effect=lambda parts: _FakeTensor(
|
||||
sum(len(p) for p in parts), 8, False, "cat"
|
||||
),
|
||||
)
|
||||
cat_patcher.start()
|
||||
self.addCleanup(cat_patcher.stop)
|
||||
# Pinned destination buffers are faked; only the copy_ call is checked.
|
||||
self.pin_patch = mock.patch.object(
|
||||
cc_mod.torch, "empty", side_effect=lambda *a, **k: _FakeTensor(8, 8, False, "pin")
|
||||
)
|
||||
self.pin_patch.start()
|
||||
self.addCleanup(self.pin_patch.stop)
|
||||
|
||||
def test_disabled_keeps_sync_path(self):
|
||||
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_SUBMIT", False):
|
||||
c = _make_controller()
|
||||
self.assertFalse(c.async_backup_enabled())
|
||||
self.assertFalse(c._async_submit_applicable())
|
||||
|
||||
def test_wrong_backend_or_layout_is_ineligible(self):
|
||||
c = _make_controller()
|
||||
c.io_backend = "kernel"
|
||||
self.assertFalse(c.async_backup_enabled())
|
||||
c = _make_controller()
|
||||
c.mem_pool_host.layout = "page_first"
|
||||
self.assertFalse(c.async_backup_enabled())
|
||||
|
||||
def test_index_targets_skip_shared_layout_and_cpu_tensors(self):
|
||||
shared = PoolName.SWA
|
||||
c = _make_controller(sidecars={shared: _FakeHostPool(shared=True)})
|
||||
transfer = PoolTransfer(
|
||||
name=shared, device_indices=_FakeTensor(4, 8, True, "swa")
|
||||
)
|
||||
cpu_anchor = _FakeTensor(8, 8, False, "anchor")
|
||||
op = _op(c, anchor=cpu_anchor, transfers=[transfer])
|
||||
self.assertEqual(c._async_index_targets(op), [])
|
||||
|
||||
c = _make_controller(sidecars={PoolName.SWA: _FakeHostPool(shared=False)})
|
||||
transfer = PoolTransfer(
|
||||
name=PoolName.SWA,
|
||||
device_indices=_FakeTensor(4, 8, True, "swa"),
|
||||
indices_from_pool=PoolName.KV,
|
||||
)
|
||||
anchor = _FakeTensor(8, 8, True, "anchor")
|
||||
op = _op(c, anchor=anchor, transfers=[transfer])
|
||||
targets = c._async_index_targets(op)
|
||||
self.assertEqual([k for k, _ in targets], [(None, None), (PoolName.SWA, PoolName.KV)])
|
||||
|
||||
def test_start_copy_orders_after_scheduler_and_records_event(self):
|
||||
log = []
|
||||
c = _make_controller()
|
||||
op = _op(c)
|
||||
with mock.patch.object(cc_mod, "device_module", _FakeCudaModule(log)):
|
||||
c._start_async_index_copy(op)
|
||||
self.assertIsNotNone(op.async_parts)
|
||||
self.assertEqual(op.async_parts.ready_event._set, True)
|
||||
self.assertEqual(op.device_indices.recorded[0].name, "indices")
|
||||
# The copy stream is ordered after the scheduler stream once.
|
||||
self.assertIn(("wait_stream", "indices", "schedule"), log)
|
||||
self.assertEqual(c._async_pinned_bytes, 64)
|
||||
|
||||
def test_pinned_cap_falls_back_to_sync(self):
|
||||
c = _make_controller()
|
||||
op = _op(c)
|
||||
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_PINNED_BYTES_MAX", 1), \
|
||||
mock.patch.object(cc_mod, "device_module", _FakeCudaModule()):
|
||||
c._start_async_index_copy(op)
|
||||
self.assertIsNone(op.async_parts)
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
|
||||
def test_materialize_uses_pool_keys(self):
|
||||
c = _make_controller()
|
||||
anchor_copy = cc_mod.AsyncIndexCopy((None, None), None, _FakeTensor(8, 8, False, "a"), 64)
|
||||
deriv = PoolTransfer(
|
||||
name=PoolName.SWA,
|
||||
device_indices=_FakeTensor(4, 8, True, "swa"),
|
||||
indices_from_pool=PoolName.KV,
|
||||
)
|
||||
side_copy = cc_mod.AsyncIndexCopy(
|
||||
(PoolName.SWA, PoolName.KV), None, _FakeTensor(4, 8, False, "s"), 32
|
||||
)
|
||||
op = _op(c, transfers=[deriv])
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
copies=[anchor_copy, side_copy], ready_event=_FakeEvent(), nbytes=96
|
||||
)
|
||||
c._materialize_async_indices([op])
|
||||
self.assertIs(op.device_indices, anchor_copy.dst)
|
||||
self.assertIs(deriv.device_indices, side_copy.dst)
|
||||
|
||||
def test_worker_waits_events_then_submits_and_acks_once(self):
|
||||
log = []
|
||||
c = _make_controller()
|
||||
events = []
|
||||
|
||||
def _make_op(tag):
|
||||
op = _op(c, anchor=_FakeTensor(8, 8, True, tag))
|
||||
ev = _FakeEvent(log, tag)
|
||||
events.append(ev)
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
copies=[
|
||||
cc_mod.AsyncIndexCopy(
|
||||
(None, None), op.device_indices, _FakeTensor(8, 8, False, tag), 64
|
||||
)
|
||||
],
|
||||
ready_event=ev,
|
||||
nbytes=64,
|
||||
)
|
||||
return op
|
||||
|
||||
ops = [_make_op("a"), _make_op("b")]
|
||||
c._async_pinned_bytes = 128
|
||||
submitted = []
|
||||
|
||||
def _submit(op):
|
||||
# Every ready event must have been waited before the merge is used.
|
||||
submitted.append(list(op.node_ids))
|
||||
return cc_mod.HiCacheAck(None, None, list(op.node_ids), num_tokens=8)
|
||||
|
||||
c._submit_write_operation = _submit
|
||||
c._write_submit_queue = cc_mod.Queue(maxsize=4)
|
||||
c._ensure_write_submit_thread = lambda: None
|
||||
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
|
||||
c._write_submit_thread = worker
|
||||
worker.start()
|
||||
try:
|
||||
item = cc_mod.WriteSubmission(ops=ops, tracer=None, enqueue_mono_ns=0)
|
||||
c._write_submit_active = 1
|
||||
c._write_submit_queue.put(item)
|
||||
deadline = time.time() + 5
|
||||
while c._write_submit_active and time.time() < deadline:
|
||||
time.sleep(0.005)
|
||||
self.assertEqual(c._write_submit_active, 0)
|
||||
finally:
|
||||
c._write_submit_queue.put(None)
|
||||
worker.join(timeout=5)
|
||||
self.assertEqual([e.waited for e in events], [1, 1])
|
||||
self.assertEqual(submitted, [[7, 7]])
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
c.drain_write_submission_results()
|
||||
self.assertEqual(len(c.ack_write_queue), 1)
|
||||
# Second drain is a no-op: exactly-once ACK.
|
||||
c.drain_write_submission_results()
|
||||
self.assertEqual(len(c.ack_write_queue), 1)
|
||||
|
||||
def test_worker_error_is_fail_closed(self):
|
||||
c = _make_controller()
|
||||
op = _op(c)
|
||||
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
|
||||
c._submit_write_operation = mock.Mock(side_effect=RuntimeError("boom"))
|
||||
c._write_submit_queue = cc_mod.Queue(maxsize=4)
|
||||
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
|
||||
c._write_submit_thread = worker
|
||||
worker.start()
|
||||
c._write_submit_active = 1
|
||||
c._write_submit_queue.put(cc_mod.WriteSubmission([op], None, 0))
|
||||
deadline = time.time() + 5
|
||||
while c._write_submit_active and time.time() < deadline:
|
||||
time.sleep(0.005)
|
||||
c._write_submit_queue.put(None)
|
||||
worker.join(timeout=5)
|
||||
with self.assertRaises(RuntimeError):
|
||||
c.drain_write_submission_results()
|
||||
self.assertEqual(c.ack_write_queue, [])
|
||||
|
||||
def test_enqueue_failure_keeps_active_consistent(self):
|
||||
c = _make_controller()
|
||||
c._write_submit_queue = cc_mod.Queue(maxsize=1)
|
||||
c._write_submit_queue.put("occupied")
|
||||
c._ensure_write_submit_thread = lambda: None
|
||||
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_ENQUEUE_TIMEOUT", 0.05):
|
||||
ok = c._enqueue_async_write([_op(c)], None)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(c._write_submit_active, 0)
|
||||
|
||||
def test_queue_full_falls_back_to_sync_submit(self):
|
||||
c = _make_controller()
|
||||
# A permanently-full submit queue forces the sync path.
|
||||
c._write_submit_queue = cc_mod.Queue(maxsize=1)
|
||||
c._write_submit_queue.put("occupied")
|
||||
c._ensure_write_submit_thread = lambda: None
|
||||
op = _op(c)
|
||||
retired_event = _FakeEvent()
|
||||
retired_event.record()
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
[cc_mod.AsyncIndexCopy((None, None), op.device_indices, _FakeTensor(8, 8, False), 64)],
|
||||
retired_event,
|
||||
64,
|
||||
)
|
||||
c._async_pinned_bytes = 64
|
||||
c.write_queue.append(op)
|
||||
submitted = []
|
||||
|
||||
def _submit(merged):
|
||||
# The fallback must hand over the original device indices, i.e. the
|
||||
# pinned parts are retired rather than materialized.
|
||||
submitted.append(merged.device_indices)
|
||||
return cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
|
||||
|
||||
c._submit_write_operation = _submit
|
||||
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_ENQUEUE_TIMEOUT", 0.05):
|
||||
c.start_writing()
|
||||
self.assertEqual(len(submitted), 1)
|
||||
self.assertIs(submitted[0], op.device_indices)
|
||||
self.assertEqual(len(c.ack_write_queue), 1)
|
||||
self.assertEqual(c.write_queue, [])
|
||||
self.assertIsNone(op.async_parts)
|
||||
self.assertEqual(c._write_submit_active, 0)
|
||||
# The retired copy is released once its event completes.
|
||||
c._reap_retired_parts()
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
|
||||
def test_mixed_queue_sync_path_retires_unconsumed_parts(self):
|
||||
c = _make_controller()
|
||||
ev = _FakeEvent()
|
||||
ev.record()
|
||||
with_parts = _op(c, node_id=1)
|
||||
with_parts.async_parts = cc_mod.AsyncIndexParts(
|
||||
[
|
||||
cc_mod.AsyncIndexCopy(
|
||||
(None, None), with_parts.device_indices, _FakeTensor(8, 8, False), 64
|
||||
)
|
||||
],
|
||||
ev,
|
||||
64,
|
||||
)
|
||||
without_parts = _op(c, node_id=2)
|
||||
c.write_queue.extend([with_parts, without_parts])
|
||||
c._async_pinned_bytes = 64
|
||||
submitted = []
|
||||
c._submit_write_operation = lambda merged: (
|
||||
submitted.append(list(merged.node_ids))
|
||||
or cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
|
||||
)
|
||||
c.start_writing()
|
||||
# Mixed residency () forces the sync path, but the pinned copy started
|
||||
# for the first op must be retired and its budget released.
|
||||
self.assertEqual(submitted, [[1, 2]])
|
||||
self.assertIsNone(with_parts.async_parts)
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
self.assertEqual(len(c.ack_write_queue), 1)
|
||||
|
||||
def test_op_cap_falls_back_to_sync(self):
|
||||
c = _make_controller()
|
||||
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_MAX_OPS", 1):
|
||||
for i in range(2):
|
||||
op = _op(c, node_id=i)
|
||||
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
|
||||
c.write_queue.append(op)
|
||||
self.assertFalse(c._async_submit_applicable())
|
||||
|
||||
def test_inflight_reflects_async_state(self):
|
||||
c = _make_controller()
|
||||
self.assertFalse(c.has_inflight_device_transfers())
|
||||
c._write_submit_active = 1
|
||||
self.assertTrue(c.has_inflight_device_transfers())
|
||||
c._write_submit_active = 0
|
||||
c._async_pinned_bytes = 8
|
||||
self.assertTrue(c.has_inflight_device_transfers())
|
||||
c._async_pinned_bytes = 0
|
||||
c._write_result_queue.put(cc_mod.WriteResult(None, [], None))
|
||||
self.assertTrue(c.has_inflight_device_transfers())
|
||||
c._write_result_queue.get()
|
||||
self.assertFalse(c.has_inflight_device_transfers())
|
||||
|
||||
def test_idle_wait_times_out(self):
|
||||
c = _make_controller()
|
||||
c._write_submit_active = 1
|
||||
with self.assertRaises(RuntimeError):
|
||||
c.wait_write_submissions_idle(timeout=0.05)
|
||||
|
||||
def test_retired_parts_release_only_after_event(self):
|
||||
c = _make_controller()
|
||||
ev = _FakeEvent()
|
||||
ev._set = False
|
||||
op = _op(c)
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
[cc_mod.AsyncIndexCopy((None, None), None, _FakeTensor(8, 8, False), 64)],
|
||||
ev,
|
||||
64,
|
||||
)
|
||||
c._async_pinned_bytes = 64
|
||||
c._retire_async_parts([op])
|
||||
self.assertIsNone(op.async_parts)
|
||||
self.assertEqual(c._async_pinned_bytes, 64)
|
||||
ev._set = True
|
||||
c._reap_retired_parts()
|
||||
self.assertEqual(c._async_pinned_bytes, 0)
|
||||
self.assertEqual(c._retire_parts, [])
|
||||
|
||||
def test_merge_of_cpu_parts_matches_cpu_reference(self):
|
||||
c = _make_controller()
|
||||
ops = []
|
||||
for i in range(3):
|
||||
op = _op(c, anchor=_FakeTensor(4, 8, True, f"a{i}"), node_id=i)
|
||||
op.async_parts = cc_mod.AsyncIndexParts(
|
||||
[
|
||||
cc_mod.AsyncIndexCopy(
|
||||
(None, None),
|
||||
op.device_indices,
|
||||
_FakeTensor(4, 8, False, f"p{i}"),
|
||||
32,
|
||||
)
|
||||
],
|
||||
_FakeEvent(),
|
||||
32,
|
||||
)
|
||||
ops.append(op)
|
||||
c._materialize_async_indices(ops)
|
||||
merged = cc_mod.CacheOperation.merge_ops(ops)
|
||||
self.assertEqual(merged.node_ids, [0, 1, 2])
|
||||
# Single-op merge must not copy: it returns the operation itself.
|
||||
single = cc_mod.CacheOperation.merge_ops([ops[0]])
|
||||
self.assertIs(single, ops[0])
|
||||
|
||||
def test_stop_waits_for_inflight_submission_and_drains_ack(self):
|
||||
c = _make_controller()
|
||||
release = threading.Event()
|
||||
started = threading.Event()
|
||||
|
||||
def _submit(op):
|
||||
started.set()
|
||||
release.wait(10)
|
||||
return cc_mod.HiCacheAck(None, None, list(op.node_ids), num_tokens=8)
|
||||
|
||||
c._submit_write_operation = _submit
|
||||
op = _op(c)
|
||||
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
|
||||
c._ensure_write_submit_thread()
|
||||
c._write_submit_active = 1
|
||||
c._write_submit_queue.put(cc_mod.WriteSubmission([op], None, 0))
|
||||
self.assertTrue(started.wait(5))
|
||||
|
||||
stopped = threading.Event()
|
||||
|
||||
def _stop():
|
||||
c._stop_write_submission_thread()
|
||||
stopped.set()
|
||||
|
||||
stopper = threading.Thread(target=_stop, daemon=True)
|
||||
stopper.start()
|
||||
# The stop must not return while a submission is still in flight.
|
||||
self.assertFalse(stopped.wait(0.05))
|
||||
release.set()
|
||||
self.assertTrue(stopped.wait(10))
|
||||
self.assertFalse(c._write_submit_thread is not None and c._write_submit_thread.is_alive())
|
||||
# The ACK produced by the in-flight submission was not dropped.
|
||||
self.assertEqual(len(c.ack_write_queue), 1)
|
||||
|
||||
def test_stop_joins_worker_and_allows_restart(self):
|
||||
c = _make_controller()
|
||||
c._ensure_write_submit_thread()
|
||||
self.assertTrue(c._write_submit_thread.is_alive())
|
||||
c._stop_write_submission_thread()
|
||||
self.assertIsNone(c._write_submit_thread)
|
||||
self.assertIsNone(c._write_submit_queue)
|
||||
c._ensure_write_submit_thread()
|
||||
self.assertTrue(c._write_submit_thread.is_alive())
|
||||
c._stop_write_submission_thread()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,504 +0,0 @@
|
||||
"""Unit tests for the SWA-only async state-payload first version.
|
||||
|
||||
Covers the v2 checklist that can run without a live server:
|
||||
|
||||
* should_async_state_payload guards (last chunk / capability / shard / CP);
|
||||
* require_async_state_event (empty last chunk / multi segment invariant);
|
||||
* worker materialize_state_indices (event ordering, no-event failure,
|
||||
non-tensor and None entries preserved);
|
||||
* kv_to_page_indices_pinned body: pinned CPU buffer, non-blocking copy,
|
||||
same dtype/shape, no synchronous .cpu();
|
||||
* a CUDA ordering test (skipped unless torch + CUDA + sglang are importable)
|
||||
proving the helper returns before a preceding long kernel finishes and
|
||||
the event-synchronized values are correct.
|
||||
|
||||
The functions are extracted from the real source with ``ast`` so the test does
|
||||
not need to import the heavy SGLang modules. Full DSV4 component / abort /
|
||||
multi-segment integration cases run in the server harness on b300.
|
||||
"""
|
||||
import ast
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
except Exception: # local helper-only runs without sglang installed
|
||||
CustomTestCase = unittest.TestCase
|
||||
|
||||
|
||||
def _repo_root():
|
||||
return pathlib.Path(__file__).resolve().parents[4]
|
||||
|
||||
|
||||
def _load_function(rel_path, name, extra_ns=None):
|
||||
path = _repo_root() / rel_path
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
fn = next(
|
||||
node
|
||||
for node in tree.body
|
||||
if isinstance(node, ast.FunctionDef) and node.name == name
|
||||
)
|
||||
namespace = dict(extra_ns or {})
|
||||
module = ast.Module(body=[fn], type_ignores=[])
|
||||
exec(compile(module, str(path), "exec"), namespace)
|
||||
return namespace[name]
|
||||
|
||||
|
||||
def _load_pinned_helper():
|
||||
"""Execute the helper body from the real source with a fake torch."""
|
||||
path = _repo_root() / "python/sglang/srt/mem_cache/common.py"
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
fn = next(
|
||||
node
|
||||
for node in tree.body
|
||||
if isinstance(node, ast.FunctionDef) and node.name == "kv_to_page_indices_pinned"
|
||||
)
|
||||
calls = {"empty": [], "copy": []}
|
||||
|
||||
class _FakeTensor:
|
||||
def __init__(self, values, dtype="torch.int32", shape=None):
|
||||
self.values = list(values)
|
||||
self.dtype = dtype
|
||||
self.shape = (len(self.values),) if shape is None else shape
|
||||
|
||||
def __getitem__(self, sl):
|
||||
return _FakeTensor(self.values[sl], self.dtype)
|
||||
|
||||
def __floordiv__(self, value):
|
||||
return _FakeTensor([v // value for v in self.values], self.dtype)
|
||||
|
||||
def cpu(self):
|
||||
raise AssertionError("kv_to_page_indices_pinned must not call .cpu()")
|
||||
|
||||
def numpy(self):
|
||||
raise AssertionError(
|
||||
"kv_to_page_indices_pinned must not call .numpy()"
|
||||
)
|
||||
|
||||
def copy_(self, src, non_blocking=False):
|
||||
calls["copy"].append((src, non_blocking))
|
||||
self.values = list(src.values)
|
||||
|
||||
class _FakeTorch:
|
||||
Tensor = _FakeTensor
|
||||
|
||||
@staticmethod
|
||||
def empty(shape, dtype, device, pin_memory):
|
||||
calls["empty"].append(
|
||||
{"shape": shape, "dtype": dtype, "device": device, "pin": pin_memory}
|
||||
)
|
||||
return _FakeTensor([], dtype=dtype, shape=shape)
|
||||
|
||||
namespace = {"torch": _FakeTorch}
|
||||
module = ast.Module(body=[fn], type_ignores=[])
|
||||
exec(compile(module, str(path), "exec"), namespace)
|
||||
return namespace["kv_to_page_indices_pinned"], calls, _FakeTensor
|
||||
|
||||
|
||||
class AsyncStateGateTest(CustomTestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.mod = types.SimpleNamespace()
|
||||
base = "python/sglang/srt/disaggregation/prefill.py"
|
||||
cls.mod.should_async_state_payload = _load_function(
|
||||
base, "should_async_state_payload"
|
||||
)
|
||||
cls.mod.require_async_state_event = _load_function(
|
||||
base, "require_async_state_event"
|
||||
)
|
||||
|
||||
def test_guards(self):
|
||||
mod = self.mod
|
||||
base = dict(
|
||||
last_chunk=True,
|
||||
supports_async_state_indices=True,
|
||||
kv_shard_size=1,
|
||||
enable_all_cp_ranks_for_transfer=False,
|
||||
enable_dsa_cache_layer_split=False,
|
||||
)
|
||||
self.assertTrue(mod.should_async_state_payload(**base))
|
||||
self.assertFalse(
|
||||
mod.should_async_state_payload(**{**base, "last_chunk": False})
|
||||
)
|
||||
self.assertFalse(
|
||||
mod.should_async_state_payload(
|
||||
**{**base, "supports_async_state_indices": False}
|
||||
)
|
||||
)
|
||||
self.assertFalse(
|
||||
mod.should_async_state_payload(**{**base, "kv_shard_size": 2})
|
||||
)
|
||||
self.assertFalse(
|
||||
mod.should_async_state_payload(
|
||||
**{**base, "enable_all_cp_ranks_for_transfer": True}
|
||||
)
|
||||
)
|
||||
self.assertTrue(
|
||||
mod.should_async_state_payload(
|
||||
**{
|
||||
**base,
|
||||
"enable_all_cp_ranks_for_transfer": True,
|
||||
"enable_dsa_cache_layer_split": True,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
def test_final_event_required(self):
|
||||
mod = self.mod
|
||||
mod.require_async_state_event(True, True, object()) # no raise
|
||||
mod.require_async_state_event(False, True, None) # sync path
|
||||
mod.require_async_state_event(True, False, None) # non-final segment
|
||||
with self.assertRaises(RuntimeError):
|
||||
mod.require_async_state_event(True, True, None)
|
||||
|
||||
|
||||
class MaterializeStateIndicesTest(CustomTestCase):
|
||||
class _FakeTensor:
|
||||
def __init__(self, values):
|
||||
self.values = values
|
||||
|
||||
def numpy(self):
|
||||
return np.array(self.values, dtype=np.int32)
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.fn = staticmethod(
|
||||
_load_function(
|
||||
"python/sglang/srt/disaggregation/mooncake/conn.py",
|
||||
"materialize_state_indices",
|
||||
)
|
||||
)
|
||||
|
||||
def _fake_torch(self):
|
||||
module = types.ModuleType("torch")
|
||||
module.Tensor = self._FakeTensor
|
||||
return module
|
||||
|
||||
def test_event_required_for_tensor(self):
|
||||
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
|
||||
with self.assertRaises(RuntimeError):
|
||||
self.fn([self._FakeTensor([1, 2])], None)
|
||||
|
||||
def test_materializes_after_event_and_preserves_others(self):
|
||||
np_indices = np.array([7, 8], dtype=np.int32)
|
||||
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
|
||||
out = self.fn([self._FakeTensor([1, 2]), None, np_indices], object())
|
||||
self.assertIsInstance(out[0], np.ndarray)
|
||||
np.testing.assert_array_equal(out[0], np.array([1, 2], dtype=np.int32))
|
||||
self.assertIsNone(out[1])
|
||||
self.assertIs(out[2], np_indices)
|
||||
|
||||
def test_second_materialization_without_event_is_safe(self):
|
||||
"""Staging retry: first pass materializes; second pass has no event."""
|
||||
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
|
||||
first = self.fn([self._FakeTensor([1, 2])], object())
|
||||
second = self.fn(first, None)
|
||||
self.assertIs(second, first)
|
||||
np.testing.assert_array_equal(first[0], np.array([1, 2], dtype=np.int32))
|
||||
|
||||
def test_no_tensor_needs_no_event(self):
|
||||
np_indices = np.array([1], dtype=np.int32)
|
||||
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
|
||||
out = self.fn([None, np_indices], None)
|
||||
self.assertIsNone(out[0])
|
||||
self.assertIs(out[1], np_indices)
|
||||
self.assertIsNone(self.fn(None, None))
|
||||
|
||||
|
||||
class PinnedHelperBodyTest(CustomTestCase):
|
||||
def test_pinned_nonblocking_no_cpu_sync(self):
|
||||
helper, calls, fake_tensor = _load_pinned_helper()
|
||||
src = fake_tensor(range(16), dtype="torch.int32")
|
||||
out = helper(src, page_size=4)
|
||||
self.assertEqual(out.values, [0, 1, 2, 3])
|
||||
self.assertEqual(out.dtype, "torch.int32")
|
||||
self.assertEqual(out.shape, (4,))
|
||||
self.assertEqual(len(calls["empty"]), 1)
|
||||
self.assertEqual(calls["empty"][0]["device"], "cpu")
|
||||
self.assertTrue(calls["empty"][0]["pin"])
|
||||
self.assertEqual(len(calls["copy"]), 1)
|
||||
self.assertTrue(calls["copy"][0][1])
|
||||
|
||||
|
||||
class CudaSendChainTest(CustomTestCase):
|
||||
"""Real _send_kv_chunk -> sender.send -> transfer_worker call chain."""
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise unittest.SkipTest("CUDA required")
|
||||
from sglang.srt.disaggregation.prefill import ( # noqa: F401
|
||||
SchedulerDisaggregationPrefillMixin,
|
||||
)
|
||||
except unittest.SkipTest:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise unittest.SkipTest(f"torch/CUDA/sglang unavailable: {exc}")
|
||||
|
||||
def test_send_chunk_queues_async_state_and_worker_materializes(self):
|
||||
import types
|
||||
|
||||
import torch
|
||||
from sglang.srt.disaggregation.base.conn import KVPoll, StateType
|
||||
from sglang.srt.disaggregation.common.utils import TransferKVChunk
|
||||
from sglang.srt.disaggregation.mooncake.conn import (
|
||||
MooncakeKVManager,
|
||||
MooncakeKVSender,
|
||||
)
|
||||
from sglang.srt.disaggregation.prefill import (
|
||||
SchedulerDisaggregationPrefillMixin,
|
||||
)
|
||||
from sglang.srt.observability import batch_telemetry # noqa: F401
|
||||
|
||||
seq_len = 32
|
||||
page_size = 8
|
||||
req_to_token = torch.arange(0, seq_len, device="cuda", dtype=torch.int32).view(
|
||||
1, seq_len
|
||||
)
|
||||
allocator = types.SimpleNamespace(
|
||||
page_size=page_size,
|
||||
get_kvcache=lambda: types.SimpleNamespace(
|
||||
request_window=None,
|
||||
sliding_window=seq_len,
|
||||
request_state_transfer_indices=lambda req_idx, length: np.array(
|
||||
[req_idx], dtype=np.int32
|
||||
),
|
||||
),
|
||||
translate_swa_indices_for_transfer=lambda x: x,
|
||||
translate_kv_indices_for_transfer=lambda x: x,
|
||||
)
|
||||
req_pool = types.SimpleNamespace(req_to_token=req_to_token)
|
||||
captured = []
|
||||
|
||||
def add_transfer_request(
|
||||
room,
|
||||
kv_indices,
|
||||
index_slice,
|
||||
is_last_chunk,
|
||||
aux_index=None,
|
||||
state_indices=None,
|
||||
num_kv_tokens=None,
|
||||
wait_event=None,
|
||||
indices_event=None,
|
||||
trace_ctx=None,
|
||||
obs=None,
|
||||
):
|
||||
chunk = TransferKVChunk(
|
||||
room,
|
||||
kv_indices,
|
||||
index_slice,
|
||||
is_last_chunk,
|
||||
aux_index,
|
||||
state_indices,
|
||||
num_kv_tokens,
|
||||
)
|
||||
chunk.indices_event = indices_event
|
||||
captured.append(chunk)
|
||||
|
||||
fake_mgr = types.SimpleNamespace(
|
||||
kv_shard_size=1,
|
||||
is_dummy_cp_rank=False,
|
||||
enable_all_cp_ranks_for_transfer=False,
|
||||
_dcp_pack_max_tokens=None,
|
||||
prepare_send_state_indices=lambda state: state,
|
||||
add_transfer_request=add_transfer_request,
|
||||
)
|
||||
sender = object.__new__(MooncakeKVSender)
|
||||
sender.kv_mgr = fake_mgr
|
||||
sender.bootstrap_room = 1
|
||||
sender.aux_index = 0
|
||||
sender.conclude_state = None
|
||||
sender.trace_ctx = mock.MagicMock()
|
||||
sender._early_send_wait_event = None
|
||||
sender._transfer_start_time = None
|
||||
sender._transfer_num_kv_indices = 0
|
||||
sender._transfer_num_state_indices = 0
|
||||
sender.num_kv_indices = seq_len // page_size
|
||||
sender.curr_idx = 0
|
||||
sender._send_kind = "last"
|
||||
|
||||
fake_sched = types.SimpleNamespace(
|
||||
token_to_kv_pool_allocator=allocator,
|
||||
req_to_token_pool=req_pool,
|
||||
disagg_metadata_buffers=types.SimpleNamespace(
|
||||
set_buf=lambda req: None
|
||||
),
|
||||
disagg_prefill_bootstrap_queue=types.SimpleNamespace(
|
||||
kv_manager=types.SimpleNamespace(
|
||||
kv_args=types.SimpleNamespace(
|
||||
state_types=[
|
||||
StateType.DSV4_REQUEST_STATE,
|
||||
StateType.REQUEST_ROW,
|
||||
StateType.SWA,
|
||||
StateType.SWA,
|
||||
]
|
||||
)
|
||||
)
|
||||
),
|
||||
disagg_prefill_pending_chunk_rids=set(),
|
||||
sliding_window_size=seq_len,
|
||||
enable_staging=False,
|
||||
)
|
||||
req = types.SimpleNamespace(
|
||||
rid="r1",
|
||||
kv=types.SimpleNamespace(req_pool_idx=0),
|
||||
start_send_idx=0,
|
||||
extend_end=seq_len,
|
||||
origin_input_ids=list(range(seq_len)),
|
||||
disagg_decode_prefix_len=0,
|
||||
pending_bootstrap=False,
|
||||
disagg_kv_sender=sender,
|
||||
metadata_buffer_index=0,
|
||||
)
|
||||
|
||||
parallel = types.SimpleNamespace(
|
||||
enable_dsa_cache_layer_split=False, num_dp_ranks=4
|
||||
)
|
||||
with mock.patch(
|
||||
"sglang.srt.disaggregation.prefill.get_parallel", lambda: parallel
|
||||
), mock.patch(
|
||||
"sglang.srt.disaggregation.common.conn.get_parallel", lambda: parallel
|
||||
):
|
||||
SchedulerDisaggregationPrefillMixin._send_kv_chunk(
|
||||
fake_sched, req, last_chunk=True
|
||||
)
|
||||
|
||||
self.assertEqual(len(captured), 1)
|
||||
chunk = captured[0]
|
||||
self.assertTrue(chunk.is_last_chunk)
|
||||
self.assertIsNotNone(chunk.indices_event)
|
||||
self.assertEqual(len(chunk.state_indices), 4)
|
||||
# DSV4_REQUEST_STATE / REQUEST_ROW stay host arrays; both SWA entries
|
||||
# are pinned CPU tensors before the worker materializes them.
|
||||
self.assertIsInstance(chunk.state_indices[0], np.ndarray)
|
||||
self.assertIsInstance(chunk.state_indices[1], np.ndarray)
|
||||
self.assertIsInstance(chunk.state_indices[2], torch.Tensor)
|
||||
self.assertIsInstance(chunk.state_indices[3], torch.Tensor)
|
||||
self.assertFalse(chunk.state_indices[2].is_cuda)
|
||||
self.assertFalse(chunk.state_indices[3].is_cuda)
|
||||
|
||||
# Now run the real Mooncake worker on the chunk produced above.
|
||||
peer = types.SimpleNamespace(
|
||||
is_dummy=False,
|
||||
mooncake_session_id="peer",
|
||||
dst_device_kv_indices=None,
|
||||
dst_kv_indices=[0, 1, 2, 3],
|
||||
required_dst_info_num=1,
|
||||
room=1,
|
||||
endpoint="127.0.0.1",
|
||||
dst_port=8998,
|
||||
dst_aux_ptrs=[],
|
||||
)
|
||||
manager = types.SimpleNamespace(
|
||||
enable_trace=False,
|
||||
enable_staging=False,
|
||||
enable_deferred_decode_kv_release=False,
|
||||
_staging_outstanding=__import__("collections").defaultdict(int),
|
||||
request_status={1: KVPoll.WaitingForInput},
|
||||
transfer_infos={1: {"peer": peer}},
|
||||
_prefill_unique_rank=lambda: 0,
|
||||
session_lock=__import__("threading").Lock(),
|
||||
failed_sessions=set(),
|
||||
state_layout_rejections={},
|
||||
poison_deferred_ack_room=lambda room: None,
|
||||
decode_kv_args_table={
|
||||
"peer": types.SimpleNamespace(
|
||||
requires_dcp_relayout=False,
|
||||
dst_kv_ptrs=[1000],
|
||||
dst_kv_layer_ids=[0],
|
||||
dst_kv_item_len=16,
|
||||
dst_attn_tp_size=1,
|
||||
dst_aux_ptrs=[],
|
||||
)
|
||||
},
|
||||
_get_dsa_cache_transfer_skip_flags=lambda reg: (False, False),
|
||||
kv_args=types.SimpleNamespace(
|
||||
kv_data_ptrs=[1], state_types=[StateType.SWA]
|
||||
),
|
||||
is_mla_backend=True,
|
||||
send_kvcache=lambda *a, **kw: 0,
|
||||
send_aux=lambda *a, **kw: 0,
|
||||
conclude_transfer=lambda *a, **kw: None,
|
||||
_maybe_ack_drained_abort=lambda room: None,
|
||||
bootstrap_port=8998,
|
||||
)
|
||||
manager.check_status = lambda room: manager.request_status[room]
|
||||
|
||||
seen = {}
|
||||
|
||||
def maybe_send_extra(req, indices, executor, registration):
|
||||
seen["types"] = [type(x) for x in indices]
|
||||
seen["swa_values"] = [indices[2].tolist(), indices[3].tolist()]
|
||||
return 0
|
||||
|
||||
manager.maybe_send_extra = maybe_send_extra
|
||||
|
||||
queue = __import__("queue").SimpleQueue()
|
||||
queue.put(chunk)
|
||||
queue.put(None)
|
||||
MooncakeKVManager.transfer_worker(manager, queue, mock.MagicMock())
|
||||
|
||||
self.assertEqual(seen["types"], [np.ndarray] * 4)
|
||||
self.assertEqual(seen["swa_values"], [[0, 1, 2, 3], [0, 1, 2, 3]])
|
||||
torch.cuda.synchronize()
|
||||
|
||||
|
||||
class CudaOrderingTest(CustomTestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise unittest.SkipTest("CUDA required")
|
||||
import sglang.srt.mem_cache.common # noqa: F401
|
||||
except unittest.SkipTest:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise unittest.SkipTest(f"torch/CUDA/sglang unavailable: {exc}")
|
||||
|
||||
def test_pinned_copy_returns_before_kernel_and_event_protects_values(self):
|
||||
import torch # noqa: F401
|
||||
from sglang.srt.mem_cache.common import (
|
||||
kv_to_page_indices,
|
||||
kv_to_page_indices_pinned,
|
||||
)
|
||||
|
||||
page_size = 8
|
||||
src = torch.arange(4096, device="cuda", dtype=torch.int32)
|
||||
# Warm up pinned allocation/CUDA context.
|
||||
_ = kv_to_page_indices_pinned(src, page_size)
|
||||
torch.cuda.synchronize()
|
||||
|
||||
stream = torch.cuda.Stream()
|
||||
with torch.cuda.stream(stream):
|
||||
torch.cuda._sleep(2_000_000_000)
|
||||
t0 = time.monotonic()
|
||||
out = kv_to_page_indices_pinned(src, page_size)
|
||||
enqueue_s = time.monotonic() - t0
|
||||
event = torch.cuda.Event()
|
||||
event.record(stream)
|
||||
|
||||
t1 = time.monotonic()
|
||||
event.synchronize()
|
||||
wait_s = time.monotonic() - t1
|
||||
|
||||
# The helper returns from the enqueue without waiting for the kernel.
|
||||
self.assertLess(enqueue_s, max(0.05, wait_s * 0.5))
|
||||
# After the event, the pinned values match the synchronous helper.
|
||||
np.testing.assert_array_equal(
|
||||
out.numpy(), kv_to_page_indices(src, page_size)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -580,109 +580,6 @@ class TestMooncakeEarlySend(CustomTestCase):
|
||||
manager.send_kvcache.assert_called_once()
|
||||
|
||||
|
||||
class TestMooncakeAsyncStateMaterialization(CustomTestCase):
|
||||
"""Real transfer_worker path for pinned state tensors (v2 async state)."""
|
||||
|
||||
def _manager(self):
|
||||
peer = SimpleNamespace(
|
||||
is_dummy=False,
|
||||
mooncake_session_id="peer",
|
||||
dst_device_kv_indices=None,
|
||||
dst_kv_indices=[10],
|
||||
required_dst_info_num=1,
|
||||
room=1,
|
||||
endpoint="127.0.0.1",
|
||||
dst_port=8998,
|
||||
dst_aux_ptrs=[],
|
||||
)
|
||||
manager = SimpleNamespace(
|
||||
enable_trace=False,
|
||||
enable_staging=False,
|
||||
enable_deferred_decode_kv_release=False,
|
||||
_staging_outstanding=defaultdict(int),
|
||||
request_status={1: KVPoll.WaitingForInput},
|
||||
transfer_infos={1: {"peer": peer}},
|
||||
_prefill_unique_rank=lambda: 0,
|
||||
session_lock=Lock(),
|
||||
failed_sessions=set(),
|
||||
state_layout_rejections={},
|
||||
poison_deferred_ack_room=MagicMock(),
|
||||
decode_kv_args_table={
|
||||
"peer": SimpleNamespace(
|
||||
requires_dcp_relayout=False,
|
||||
dst_kv_ptrs=[1000],
|
||||
dst_kv_layer_ids=[0],
|
||||
dst_kv_item_len=16,
|
||||
dst_attn_tp_size=1,
|
||||
dst_aux_ptrs=[],
|
||||
)
|
||||
},
|
||||
_get_dsa_cache_transfer_skip_flags=lambda reg: (False, False),
|
||||
kv_args=SimpleNamespace(kv_data_ptrs=[1], state_types=[StateType.SWA]),
|
||||
is_mla_backend=True,
|
||||
send_kvcache=MagicMock(return_value=0),
|
||||
send_aux=MagicMock(return_value=0),
|
||||
maybe_send_extra=MagicMock(return_value=0),
|
||||
conclude_transfer=MagicMock(),
|
||||
_maybe_ack_drained_abort=MagicMock(),
|
||||
bootstrap_port=8998,
|
||||
)
|
||||
manager.check_status = lambda room: manager.request_status[room]
|
||||
return manager
|
||||
|
||||
def _run(self, manager, chunk):
|
||||
queue = SimpleQueue()
|
||||
queue.put(chunk)
|
||||
queue.put(None)
|
||||
MooncakeKVManager.transfer_worker(manager, queue, MagicMock())
|
||||
|
||||
def test_worker_materializes_state_after_event(self):
|
||||
manager = self._manager()
|
||||
state = torch.tensor([0], dtype=torch.int32)
|
||||
event = MagicMock()
|
||||
order = []
|
||||
|
||||
def sync():
|
||||
order.append("sync")
|
||||
state.fill_(7)
|
||||
|
||||
event.synchronize.side_effect = sync
|
||||
chunk = TransferKVChunk(
|
||||
1, np.array([0], dtype=np.int32), slice(0, 1), True, 0, [state]
|
||||
)
|
||||
chunk.indices_event = event
|
||||
|
||||
def maybe_send_extra(req, indices, executor, registration):
|
||||
order.append("state")
|
||||
self.assertIsInstance(indices[0], np.ndarray)
|
||||
self.assertEqual(indices[0].tolist(), [7])
|
||||
return 0
|
||||
|
||||
manager.maybe_send_extra.side_effect = maybe_send_extra
|
||||
self._run(manager, chunk)
|
||||
|
||||
self.assertEqual(order, ["sync", "state"])
|
||||
self.assertIsInstance(chunk.state_indices[0], np.ndarray)
|
||||
manager.send_aux.assert_called_once()
|
||||
|
||||
def test_worker_rejects_state_tensor_without_event(self):
|
||||
manager = self._manager()
|
||||
chunk = TransferKVChunk(
|
||||
1,
|
||||
np.array([0], dtype=np.int32),
|
||||
slice(0, 1),
|
||||
True,
|
||||
0,
|
||||
[torch.tensor([0], dtype=torch.int32)],
|
||||
)
|
||||
with self.assertRaises(RuntimeError) as ctx:
|
||||
self._run(manager, chunk)
|
||||
self.assertIn(
|
||||
"async state indices require an indices_event", str(ctx.exception)
|
||||
)
|
||||
manager.maybe_send_extra.assert_not_called()
|
||||
|
||||
|
||||
class TestMooncakeStagingDeviceType(unittest.TestCase):
|
||||
"""Staging init must receive the runtime's device, not a hardcoded "cuda".
|
||||
|
||||
|
||||
@@ -31,7 +31,6 @@ register_cpu_ci(est_time=11, suite="base-a-test-cpu")
|
||||
|
||||
class _FakeOpenAIServingChat:
|
||||
native_reasoning_history = False
|
||||
native_tool_loading = False
|
||||
|
||||
def __init__(self, stream_lines=None, chat_template=None):
|
||||
self.stream_lines = stream_lines or []
|
||||
@@ -62,7 +61,6 @@ class _FakeNonStreamingErrorOpenAI:
|
||||
"""Returns an error response or raises an exception from the OpenAI handler."""
|
||||
|
||||
supports_inline_system = False
|
||||
native_tool_loading = False
|
||||
|
||||
def __init__(self, status_code=400, body=None, content=None, exception=None):
|
||||
self._status_code = status_code
|
||||
@@ -498,26 +496,6 @@ class TestAnthropicServing(unittest.TestCase):
|
||||
],
|
||||
)
|
||||
|
||||
def test_tool_addition_fails_where_tools_cannot_load_mid_conversation(self):
|
||||
"""A tool_addition block must not vanish on a model that cannot render it."""
|
||||
request = self._anthropic_request(
|
||||
messages=[
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "system",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_addition",
|
||||
"tool": {"type": "tool_reference", "name": "Bash"},
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "tool_addition"):
|
||||
self._serving()._convert_to_chat_completion_request(request)
|
||||
|
||||
def test_builtin_web_search_tool_without_schema_is_skipped(self):
|
||||
request = AnthropicMessagesRequest.model_validate(
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,181 @@
|
||||
"""Decoder SWA bounded replay with SGLANG_OPT_DSV41_PROMPT_WINDOW_TAIL: a prefill
|
||||
chunk runs the late layers over its rows in the prompt's last window only, so any
|
||||
chunking of a prompt computes exactly the rows and floors of the unchunked prompt
|
||||
there, plus one placeholder row per chunk outside it. CPU only."""
|
||||
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.attention.dsv4.attn_metadata_kernels import (
|
||||
late_layer_tail_layout,
|
||||
late_layer_tail_lens,
|
||||
)
|
||||
from sglang.srt.layers.attention.deepseek_v4_backend import (
|
||||
SWA_WINDOW,
|
||||
late_layer_num_tokens,
|
||||
)
|
||||
from sglang.srt.model_executor.forward_batch_info import (
|
||||
CaptureHiddenMode,
|
||||
ForwardMode,
|
||||
prompt_rest_lens_of,
|
||||
)
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
|
||||
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
|
||||
|
||||
CPU = torch.device("cpu")
|
||||
|
||||
|
||||
def _layout(extend_lens, seq_lens, rest=None, whole_extend=None):
|
||||
return late_layer_tail_layout(
|
||||
extend_lens_cpu=extend_lens,
|
||||
seq_lens_cpu=seq_lens,
|
||||
tail_len=SWA_WINDOW,
|
||||
device=CPU,
|
||||
whole_extend=whole_extend,
|
||||
prompt_rest_lens=rest,
|
||||
)
|
||||
|
||||
|
||||
def _chunked_rows(prompt_len, chunks):
|
||||
"""(absolute position, floor) of every tail row over a prompt's chunks, each
|
||||
chunk one single-request step."""
|
||||
rows, start = [], 0
|
||||
for n in chunks:
|
||||
end = start + n
|
||||
token_indices, _, floor = _layout([n], [end], rest=[prompt_len - end])
|
||||
rows += list(zip((start + token_indices).tolist(), floor.tolist()))
|
||||
start = end
|
||||
assert start == prompt_len
|
||||
return rows
|
||||
|
||||
|
||||
class TestTailLens(CustomTestCase):
|
||||
def test_final_chunks_keep_the_window_tail(self):
|
||||
extend_lens = [300, 50, 128]
|
||||
self.assertEqual(
|
||||
late_layer_tail_lens(extend_lens, SWA_WINDOW, prompt_rest_lens=[0, 0, 0]),
|
||||
late_layer_tail_lens(extend_lens, SWA_WINDOW),
|
||||
)
|
||||
|
||||
def test_earlier_chunks_keep_their_rows_in_the_prompt_window(self):
|
||||
# Far from the end: one row. Straddling the window: the rows inside it.
|
||||
# Inside the window: the whole chunk.
|
||||
self.assertEqual(
|
||||
late_layer_tail_lens(
|
||||
[32768, 300, 40, 300],
|
||||
SWA_WINDOW,
|
||||
prompt_rest_lens=[5000, 50, 20, SWA_WINDOW],
|
||||
),
|
||||
[1, SWA_WINDOW - 50, 40, 1],
|
||||
)
|
||||
|
||||
def test_whole_extend_wins(self):
|
||||
self.assertEqual(
|
||||
late_layer_tail_lens(
|
||||
[300, 300], SWA_WINDOW, [True, False], prompt_rest_lens=[5000, 5000]
|
||||
),
|
||||
[300, SWA_WINDOW],
|
||||
)
|
||||
# No whole request: the prompt window applies, as in the layout.
|
||||
self.assertEqual(
|
||||
late_layer_tail_lens([300], SWA_WINDOW, [False], prompt_rest_lens=[5000]),
|
||||
[1],
|
||||
)
|
||||
|
||||
|
||||
class TestTailLayout(CustomTestCase):
|
||||
def test_final_chunks_match_the_window_layout(self):
|
||||
# Negative control: final chunks (nothing left) change no row or floor.
|
||||
for extend_lens, seq_lens in (([300], [900]), ([300, 50, 40], [900, 50, 400])):
|
||||
with self.subTest(extend_lens=extend_lens):
|
||||
got = _layout(extend_lens, seq_lens, rest=[0] * len(extend_lens))
|
||||
want = _layout(extend_lens, seq_lens)
|
||||
torch.testing.assert_close(got[0], want[0])
|
||||
self.assertEqual(got[1], want[1])
|
||||
torch.testing.assert_close(got[2], want[2])
|
||||
|
||||
def test_placeholder_row_reads_only_itself(self):
|
||||
token_indices, tail_lens, floor = _layout([32768], [65536], rest=[70000])
|
||||
self.assertEqual(tail_lens, [1])
|
||||
self.assertEqual(token_indices.tolist(), [32767])
|
||||
self.assertEqual(floor.tolist(), [65535])
|
||||
|
||||
def test_batched_layout_matches_single_requests(self):
|
||||
extend_lens = [32768, 300, 40, 200]
|
||||
seq_lens = [40000, 900, 60, 200]
|
||||
rest = [9000, 50, 20, 0]
|
||||
token_indices, tail_lens, floor = _layout(extend_lens, seq_lens, rest)
|
||||
want_indices, want_floor, start = [], [], 0
|
||||
for n, s, r in zip(extend_lens, seq_lens, rest):
|
||||
idx, _, fl = _layout([n], [s], rest=[r])
|
||||
want_indices.append(start + idx)
|
||||
want_floor.append(fl)
|
||||
start += n
|
||||
self.assertEqual(tail_lens, [1, SWA_WINDOW - 50, 40, SWA_WINDOW])
|
||||
torch.testing.assert_close(token_indices, torch.cat(want_indices))
|
||||
torch.testing.assert_close(floor, torch.cat(want_floor))
|
||||
|
||||
|
||||
class TestChunkingInvariance(CustomTestCase):
|
||||
"""The rows in the prompt's last window, and the floor each reads from, do
|
||||
not depend on how the prompt was chunked."""
|
||||
|
||||
def test_any_chunking_computes_the_unchunked_window(self):
|
||||
prompt_len = 1000
|
||||
whole = _chunked_rows(prompt_len, [prompt_len])
|
||||
floor = prompt_len - SWA_WINDOW
|
||||
self.assertEqual(whole, [(p, floor) for p in range(floor, prompt_len)])
|
||||
for chunks in (
|
||||
[400, 400, 200],
|
||||
[900, 60, 40],
|
||||
[800, 100, 60, 40],
|
||||
[872, 1, 127],
|
||||
[700, 200, 100],
|
||||
):
|
||||
with self.subTest(chunks=chunks):
|
||||
rows = _chunked_rows(prompt_len, chunks)
|
||||
window = [(p, f) for p, f in rows if p >= floor]
|
||||
self.assertEqual(window, whole)
|
||||
# Placeholders sit below the window and read only their own row.
|
||||
for p, f in rows:
|
||||
if p < floor:
|
||||
self.assertEqual(f, p)
|
||||
|
||||
def test_window_rows_never_read_below_the_window(self):
|
||||
rows = _chunked_rows(1000, [900, 60, 40])
|
||||
self.assertTrue(all(f == 1000 - SWA_WINDOW for p, f in rows if p >= 872))
|
||||
|
||||
|
||||
class TestPromptRestLens(CustomTestCase):
|
||||
def test_rest_from_the_batch_ends(self):
|
||||
prompt = SimpleNamespace(full_untruncated_fill_ids=list(range(1000)))
|
||||
decode = SimpleNamespace(full_untruncated_fill_ids=list(range(70)))
|
||||
# A chunk ending at 400, a replay batch ending at its prefix 128, and a
|
||||
# decode row carried in the extend.
|
||||
self.assertEqual(
|
||||
prompt_rest_lens_of([prompt, prompt, decode], [400, 128, 70], {id(decode)}),
|
||||
[600, 872, 0],
|
||||
)
|
||||
|
||||
def test_rows_counted_for_dp_match_the_layout(self):
|
||||
extend_lens, rest = [32768, 300, 40], [9000, 50, 20]
|
||||
fb = SimpleNamespace(
|
||||
forward_mode=ForwardMode.EXTEND,
|
||||
extend_seq_lens_cpu=extend_lens,
|
||||
extend_logprob_start_lens_cpu=None,
|
||||
return_logprob=False,
|
||||
capture_hidden_mode=CaptureHiddenMode.FULL,
|
||||
extend_prompt_rest_lens_cpu=rest,
|
||||
)
|
||||
_, tail_lens, _ = _layout(extend_lens, [40000, 900, 60], rest)
|
||||
self.assertEqual(late_layer_num_tokens(fb, dspark=True), sum(tail_lens))
|
||||
fb.extend_prompt_rest_lens_cpu = None
|
||||
self.assertEqual(late_layer_num_tokens(fb, dspark=True), 2 * SWA_WINDOW + 40)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -265,27 +265,6 @@ class TestDPBudgetUpdateBudget(CustomTestCase):
|
||||
)
|
||||
self.assertEqual(budget.total_tokens, [150, 80])
|
||||
|
||||
def test_maps_waiting_uncached_tokens_and_hit_rate(self):
|
||||
budget = DPBudget(num_dp_ranks=2)
|
||||
budget.update_budget(
|
||||
[
|
||||
_load(
|
||||
dp_rank=0,
|
||||
timestamp=1.0,
|
||||
num_waiting_uncached_tokens=900,
|
||||
cache_hit_rate=0.95,
|
||||
),
|
||||
_load(
|
||||
dp_rank=1,
|
||||
timestamp=1.0,
|
||||
num_waiting_uncached_tokens=100,
|
||||
cache_hit_rate=0.5,
|
||||
),
|
||||
]
|
||||
)
|
||||
self.assertEqual(budget.total_uncached_tokens, [900, 100])
|
||||
self.assertEqual(budget.cache_hit_rate, [0.95, 0.5])
|
||||
|
||||
def test_partial_update_only_affects_reported_rank(self):
|
||||
budget = DPBudget(num_dp_ranks=3)
|
||||
budget.update_budget(
|
||||
@@ -494,44 +473,6 @@ class TestTotalRequestsScheduler(CustomTestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestUncachedTokensScheduler(CustomTestCase):
|
||||
def test_dispatches_to_rank_with_least_uncached_work(self):
|
||||
ctl = _make_controller(num_dp_ranks=4)
|
||||
ctl.dp_budget.total_uncached_tokens = [100, 50, 900, 10]
|
||||
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 1000))
|
||||
ctl.workers[3].send_pyobj.assert_called_once()
|
||||
for i in (0, 1, 2):
|
||||
ctl.workers[i].send_pyobj.assert_not_called()
|
||||
# No hit rate reported -> the whole prompt is charged as uncached.
|
||||
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [100, 50, 900, 1010])
|
||||
self.assertEqual(ctl.dp_budget.total_requests, [0, 0, 0, 1])
|
||||
|
||||
def test_charge_uses_fleet_average_miss_rate(self):
|
||||
ctl = _make_controller(num_dp_ranks=2)
|
||||
ctl.dp_budget.total_uncached_tokens = [0, 0]
|
||||
# Fleet-average hit rate 0.5 -> every candidate charges 500 tokens, so
|
||||
# the pick is decided by queued work alone (no per-rank feedback).
|
||||
ctl.dp_budget.cache_hit_rate = [0.9, 0.1]
|
||||
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 1000))
|
||||
ctl.workers[0].send_pyobj.assert_called_once()
|
||||
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [500, 0])
|
||||
|
||||
def test_pending_backlog_beats_a_cheaper_candidate(self):
|
||||
ctl = _make_controller(num_dp_ranks=2)
|
||||
ctl.dp_budget.total_uncached_tokens = [500, 0]
|
||||
ctl.dp_budget.cache_hit_rate = [0.0, 0.0]
|
||||
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 100))
|
||||
ctl.workers[1].send_pyobj.assert_called_once()
|
||||
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [500, 100])
|
||||
|
||||
def test_routed_dp_rank_bypasses_budget(self):
|
||||
ctl = _make_controller(num_dp_ranks=4)
|
||||
ctl.dp_budget.total_uncached_tokens = [5, 3, 1, 4]
|
||||
ctl.total_uncached_tokens_scheduler(_req(routed_dp_rank=0))
|
||||
ctl.workers[0].send_pyobj.assert_called_once()
|
||||
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [5, 3, 1, 4])
|
||||
|
||||
|
||||
class TestStatusAwarenessInconsistency(CustomTestCase):
|
||||
"""Document a divergence: ``round_robin_scheduler`` skips workers whose
|
||||
``status`` is False, but ``total_requests_scheduler`` /
|
||||
|
||||
@@ -510,7 +510,6 @@ class TestSchedulerIdleStepCounters(CustomTestCase):
|
||||
recv_requests=Mock(side_effect=[[]] * len(schedule) + [StopIteration])
|
||||
)
|
||||
scheduler.process_input_requests = Mock()
|
||||
scheduler.load_snapshot_writer = None
|
||||
scheduler.disagg_prefill_bootstrap_queue = SimpleNamespace(
|
||||
pop_bootstrapped=Mock(return_value=[])
|
||||
)
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user