Compare commits
22
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8b67636010 | ||
|
|
310ad35430 | ||
|
|
bdbd8dffb8 | ||
|
|
f363a6ab7b | ||
|
|
e4c7e540be | ||
|
|
b8b205b367 | ||
|
|
b3d6d19c7d | ||
|
|
8f53c1fb73 | ||
|
|
42f228bfb8 | ||
|
|
242a7ad23a | ||
|
|
2eb163ad73 | ||
|
|
d3240c745e | ||
|
|
433257c7b0 | ||
|
|
7980a09c0a | ||
|
|
a7c668a936 | ||
|
|
0c6849e378 | ||
|
|
85605b8c72 | ||
|
|
b6a5d70cba | ||
|
|
cac60f7c2e | ||
|
|
68c2af7957 | ||
|
|
497aeff5f6 | ||
|
|
5fa8c23227 |
@@ -0,0 +1,25 @@
|
||||
# L3 SSD image: recipe-ns engine + official mooncake 0.3.14 (CUDA 13, cp312).
|
||||
#
|
||||
# Build context: this directory plus the 0.3.14 wheel
|
||||
# mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
# Use build-l3ssd-image.sh, which downloads and verifies the wheel.
|
||||
#
|
||||
# Base tag recipe-ns-433257c7b-baked-20261114 = b300/integration @ 433257c7b
|
||||
# (recipe namespace/chat-ns + gateway binary baked). Rebuild it from that commit
|
||||
# if the local tag is missing.
|
||||
FROM sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-433257c7b-baked-20261114
|
||||
|
||||
COPY mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/
|
||||
RUN python3 -m pip install --no-deps --force-reinstall \
|
||||
/tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
|
||||
&& rm -f /tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
|
||||
# The wheel's master binary uses $ORIGIN/../mooncake_transfer_engine_cuda13.libs;
|
||||
# keep it in place and expose a wrapper at the legacy path used by the manifests.
|
||||
&& printf '%s\n' \
|
||||
'#!/bin/sh' \
|
||||
'export LD_LIBRARY_PATH=/opt/sglang/lib/python3.12/site-packages/mooncake_transfer_engine_cuda13.libs:$LD_LIBRARY_PATH' \
|
||||
'exec /opt/sglang/lib/python3.12/site-packages/mooncake/mooncake_master "$@"' \
|
||||
> /opt/sglang/bin/mooncake_master \
|
||||
&& chmod +x /opt/sglang/bin/mooncake_master \
|
||||
&& python3 -c "import importlib.metadata as m; print('mooncake:', m.version('mooncake-transfer-engine-cuda13'))" \
|
||||
&& /opt/sglang/bin/mooncake_master --version | head -1
|
||||
@@ -0,0 +1,139 @@
|
||||
# B300 L3 Mooncake SSD 基线部署(2026-11-15)
|
||||
|
||||
> 以同事在 `ds41-gist-exp` 上验证的 **L3 Mooncake SSD** 配置为基础,使用我们的
|
||||
> `b300/integration` 引擎代码 + 官方 **mooncake 0.3.14**,不再依赖 hostPath
|
||||
> `/opt/mooncake-latest`。
|
||||
|
||||
## 1. 版本要求(重要)
|
||||
|
||||
- **mooncake `0.3.14`**(`mooncake-transfer-engine-cuda13`,CUDA 13 / cp312 wheel)。
|
||||
- 镜像内原为 `0.3.13.post1`;0.3.14 含上游 SSD offload 相关修复/增强。
|
||||
- wheel 与 sha256 见 `requirements-mooncake.txt`,安装必须 `--no-deps`(基底已有运行时依赖)。
|
||||
- 引擎:`b300/integration` @ `433257c7b`(recipe namespace/chat-ns + 新 gateway)。
|
||||
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
|
||||
(40.1GB,已 `ctr -n k8s.io images import`;本地 docker tag `ymkymx/sglang:...` 同名)。
|
||||
- 构建/装载:`./build-l3ssd-image.sh`(下载校验 wheel → docker build → ctr import)。
|
||||
|
||||
镜像内 `/opt/sglang/bin/mooncake_master` 是 wrapper:设置 wheel `.libs` 的
|
||||
`LD_LIBRARY_PATH` 后 exec 包内 master(已验证 `0.3.14 (git e1ea5d3)`)。
|
||||
|
||||
## 2. L3 SSD 配置
|
||||
|
||||
`mooncake-store-l3ssd.json`:
|
||||
|
||||
```json
|
||||
{"master_server_address":"127.0.0.1:50051","metadata_server":"P2PHANDSHAKE",
|
||||
"global_segment_size":"200gb","protocol":"tcp",
|
||||
"enable_ssd_offload":true,"ssd_offload_path":"/data/ymk/mooncake-ssd"}
|
||||
```
|
||||
|
||||
- 部署时把该文件放到节点 `/data/ymk/ds41/mooncake-store-l3ssd.json`,SSD 目录
|
||||
`/data/ymk/mooncake-ssd` 需存在。
|
||||
- `MOONCAKE_OFFLOAD_*`:`bucket_storage_backend` / 512GiB bucket 上限 / `fifo` /
|
||||
`flush_partial=0`(见 manifest)。
|
||||
- master 参数必须带
|
||||
`--enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false`。
|
||||
|
||||
## 3. manifest
|
||||
|
||||
| 文件 | 用途 |
|
||||
|---|---|
|
||||
| `ds41-l3ssd-test-baked-20261115.yaml` | 新建 cw `ds41-l3ssd-test`(推荐先做基线验证) |
|
||||
| `ds41-gist-exp-l3ssd-baked-20261115.yaml` | 原名 `ds41-gist-exp`,用于直接更新原栈 |
|
||||
|
||||
已通过 `kmctl create/update --dry-run` 校验。启动前先停掉占用 8 卡的 stack
|
||||
(`ds41-recipe-ns-test` 或旧 `ds41-gist-exp`),确认 GPU 归零:
|
||||
|
||||
```bash
|
||||
kmctl create cw ds41-l3ssd-test --node-group B300 \
|
||||
-f deploy/b300-k8s/ds41-l3ssd-test-baked-20261115.yaml
|
||||
# 或替换原栈:
|
||||
# kmctl update cw ds41-gist-exp -f deploy/b300-k8s/ds41-gist-exp-l3ssd-baked-20261115.yaml
|
||||
```
|
||||
|
||||
端口/角色与旧 `ds41-gist-exp` 一致:master 50051、P 30000、D 30300、gateway 30400。
|
||||
启动约 10-15 分钟;核对 P/D `max_total_num_tokens` 和 master 日志中的 SSD offload 初始化。
|
||||
|
||||
## 4. 与上一版配置的差异
|
||||
|
||||
P/D 的 `sglang serve` 参数**完全一致**;差异只在 mooncake:
|
||||
|
||||
| 项 | 旧(11-09) | 本基线 |
|
||||
|---|---|---|
|
||||
| mooncake | 镜像内 0.3.13.post1 | 镜像内 **0.3.14** |
|
||||
| master | `/opt/sglang/bin/mooncake_master`,无 offload | 同路径(0.3.14 wrapper),带 offload 参数 |
|
||||
| P/D PYTHONPATH | 无 | 无(0.3.14 已烘入) |
|
||||
| L3 配置 | 纯 TCP,无 SSD | `enable_ssd_offload` + `ssd_offload_path` |
|
||||
| `MOONCAKE_OFFLOAD_*` | 无 | bucket backend / 512GiB / fifo / flush_partial=0 |
|
||||
| TMPDIR/XDG/HF | 默认/`/data/ymk` | `/data/ymk/l3ssd/{master,p,d}-*` |
|
||||
|
||||
## 5. 注意
|
||||
|
||||
- 同事原配置使用 `SGLANG_TRACE_LOOP_SAMPLE=1.0`(全量 detail tracing,约 2% 吞吐 +
|
||||
p95 开销)。本基线为对齐其配置保留 1.0;正式上线建议改回 `0.05`。
|
||||
- `global_segment_size=200gb` + 512GiB SSD offload:上线前按宿主内存总账复核。
|
||||
- SSD 目录增长需监控(`/data` 当前约 7.6T 可用)。
|
||||
- 旧镜像 `recipe-ns-433257c7b-baked-20261114` 与 `recipe-ns-a7c668a93-baked-20261114`
|
||||
保留作回滚/对照。
|
||||
|
||||
## 6. Tokenizer A/B(2026-11-15,c580 / 1200s)
|
||||
|
||||
同一 `ds41-gist-exp`(L3 SSD + mooncake 0.3.14)整栈重启,只切 P/D
|
||||
`--tokenizer-backend`;AgentX c580 / 1200s / warmup 256 / seed 42,经 30400。
|
||||
|
||||
| 臂 | tokenizer | rps | in tok/s | out tok/s | TTFT p50/p95/p99 | ITL p50/p95/p99 | requests |
|
||||
|---|---|---:|---:|---:|---|---|---:|
|
||||
| A | `huggingface`(默认) | 23.789 | 2,257,661 | 22,224 | 2.043 / 4.176 / 7.579 s | 8.02 / 14.48 / 20.91 ms | 29,498 |
|
||||
| B | **`fastokens`** | **24.045** | **2,289,885** | **22,475** | **1.746 / 3.676 / 6.820 s** | 8.13 / 11.65 / 14.87 ms | 29,816 |
|
||||
|
||||
- 结论:fastokens 臂 **rps +1.1%**、TTFT p50/p95/p99 **−14.5% / −12.0% / −10.0%**;
|
||||
ITL p95 的差异大概率是 run-to-run 噪声(decode 不走 tokenizer),不作为判据。
|
||||
- #48928 的分段编码已生效:容器内 `DeepSeekRecipe.from_hf_tokenizer(..., fastokens tokenizer)`
|
||||
的 `_fast_prompt_tokenizer` 非 None(`fastokens.Tokenizer`);服务端未打印
|
||||
“fastokens segmented” 日志,属日志路径差异,不影响结论。
|
||||
- **当前线上实验栈保留 fastokens 臂**;要回默认用
|
||||
`ds41-gist-exp-l3ssd-baked-20261115.yaml` 更新并整栈重启。
|
||||
- 单轮对比,c580 本身有方差;如需正式定论建议每臂 ≥3 轮。
|
||||
- SSD offload 已配置(master 报 `SSD Storage: 0 B / 2.00 TB`),本次 c580 工作集未触发
|
||||
SSD 逐出(`/data/ymk/mooncake-ssd` 约 20K),不是配置未生效。
|
||||
|
||||
## 7. 多轮验证(2026-11-15,每轮整栈重启)
|
||||
|
||||
> 之前 B2/B3 在同一实例上连跑、未重启,数据受实例状态影响;下面只列**整栈重启后**的轮次。
|
||||
> 整栈重启 = `kmctl stop` + `update`(换臂),或同臂时删除 p/d/gateway/master 四个 Pod。
|
||||
|
||||
| 臂 | 轮次 | rps | TTFT p50/p95/p99 (s) | ITL p95 (ms) | requests |
|
||||
|---|---|---:|---|---:|---:|
|
||||
| A(huggingface) | A1 | 23.789 | 2.043 / 4.176 / 7.579 | 14.48 | 29,498 |
|
||||
| A | A2 | 23.552 | 2.305 / 5.894 / 8.600 | 15.05 | 29,205 |
|
||||
| B(fastokens) | B1 | 24.045 | 1.746 / 3.676 / 6.820 | 11.65 | 29,816 |
|
||||
| B | **B2(fresh)** | **24.040** | **1.708 / 3.451 / 7.105** | 11.71 | 29,810 |
|
||||
|
||||
- **B2 满足判据:TTFT p95 = 3.451s < 5s**,且与 B1(3.676s)一致;
|
||||
A 臂 fresh 轮 p95 为 4.176 / 5.894s。
|
||||
- fresh 轮均值:A rps 23.67 / p95 5.04s;B rps 24.04 / p95 3.56s
|
||||
(B +1.6% rps、p95 −29%)。
|
||||
- 结论:保留 fastokens;之前未重启的 B2/B3(p95 5.99/6.69s)仅作历史参考、不纳入。
|
||||
|
||||
## 8. L3 SSD 4TB + offload_on_evict=false(2026-11-15)
|
||||
|
||||
- 启动清单 `ds41-gist-exp-l3ssd-fastokens-20261115.yaml` 已改:master
|
||||
`--offload_on_evict=false`;P 的 `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` /
|
||||
`MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` 改为 1 TiB/rank,4 rank 合计
|
||||
master 报 `SSD Storage: 0 B / 4.00 TB`。
|
||||
- 每次实验前必须重置 L3:stop cw → 确认 Terminated 0/0 + GPU 0 MiB →
|
||||
`bash clean_l3ssd_cache.sh`(删除 `/data/ymk/mooncake-ssd` 下的 rank_*)→ update/start。
|
||||
本次清理释放 1.2T,`/data` 可用 6.4T→7.5T。
|
||||
- c580(1200s,warmup 256,seed 42,fastokens,干净 L3):
|
||||
rps 24.003、in 2.285M tok/s、out 22,447、TTFT p50/p95/p99 1.768/3.500/7.549s、
|
||||
ITL 8.15/11.74/15.31ms、29,764 请求、0 错误、valid。
|
||||
对比 B1/B2:24.045/1.746/3.676/6.820 与 24.040/1.708/3.451/7.105——
|
||||
吞吐与尾延迟均在方差内,**没有可归因于 offload_on_evict/4TB/清理的收益或回归**。
|
||||
- **重要:`offload_on_evict=false` 并没有停止 SSD 写入**。run 后 master
|
||||
`SSD Storage: 533.81 GB / 4.00 TB`,`du /data/ymk/mooncake-ssd = 535G`;
|
||||
这是 `--hicache-write-policy write_through` 的 backup 写入,和内存 eviction 无关。
|
||||
若要真正避免 SSD 写入,需要改 `write_back` 或关闭 `--enable_offload`,但那会改变
|
||||
L3 语义,需另做 A/B。
|
||||
- run 后 P cgroup `memory.current=2,198,904,565,760` ≈ 2047.5 GiB,已顶到 2 TiB
|
||||
limit;宿主 MemAvailable 1.2 TiB。下一次实验前必须先停/清/重启;不要在同一实例
|
||||
上连续跑,否则 L3 与 cgroup 都处于顶格状态。
|
||||
@@ -0,0 +1,10 @@
|
||||
# 2026-11-17 c580 基线:D tokenizer 16 + fastokens
|
||||
|
||||
- manifest: `manifest.yaml`(= `ds41-gist-exp-20261117-dtok16-fastokens.yaml`)
|
||||
- P: cache / 无 optimistic / tokenizer-worker 16 / fastokens / hicache ratio 4 / L3 SSD 4T
|
||||
- D: tokenizer-worker 16 / fastokens
|
||||
- Gateway: 正常(无 P admission cap,health/circuit 开启,decode-max-running 512)
|
||||
- L3:每轮前清空;完整重启
|
||||
- c580 结果:rps 24.045,in 2.290M tok/s,out 22.449k tok/s,
|
||||
TTFT 1.743/3.772/6.580 s,ITL p95 10.74 ms,29,816 req,valid=true。
|
||||
- 原始结果:`experiments/agentx-align/results-c580-dtok16-fastokens-20261117/`
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1 @@
|
||||
{"rps": 24.04512015569562, "in_tput": 2290117.7612668574, "out_tput": 22449.32127276801, "ttft_p50": 1743.335193, "ttft_p95": 3771.88552025, "ttft_p99": 6579.86958935, "itl_p50": 8.162268671611512, "itl_p95": 10.741317697249674, "itl_p99": 11.806237803568294, "lat_p95": 31884.86991075, "requests": 29816.0, "isl_p50": 79909.0, "osl_p50": 339.0, "cache_pct": NaN, "valid": true}
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build and load the L3 SSD image (mooncake 0.3.14) on b300-01.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
WHEEL=mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
URL=https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/$WHEEL
|
||||
SHA=74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
|
||||
TAG=recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
REPO=sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang
|
||||
|
||||
if [ ! -f "$WHEEL" ]; then
|
||||
curl -fL -o "$WHEEL" "$URL"
|
||||
fi
|
||||
echo "$SHA $WHEEL" | sha256sum -c -
|
||||
|
||||
docker build -t "ymkymx/sglang:$TAG" -f Dockerfile.mooncake-0.3.14 .
|
||||
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
|
||||
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
|
||||
echo "loaded $REPO:$TAG"
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Reset the mooncake L3 SSD KV cache before an experiment.
|
||||
#
|
||||
# MUST run on b300-01 only after the ds41-gist-exp workload is fully stopped:
|
||||
# 1) kmctl stop cw ds41-gist-exp --wait ...
|
||||
# 2) kubectl get cw ... -> Terminated 0/0, nvidia-smi -> 0 MiB on GPU0-7
|
||||
# 3) bash clean_l3ssd_cache.sh
|
||||
# 4) kmctl update/start the new manifest
|
||||
#
|
||||
# The script refuses to delete anything while a mooncake master or sglang
|
||||
# server process is still visible on the host.
|
||||
set -euo pipefail
|
||||
|
||||
SSD_DIR="${SSD_DIR:-/data/ymk/mooncake-ssd}"
|
||||
|
||||
if pgrep -af '[m]ooncake_master|[s]glang serve' >/dev/null 2>&1; then
|
||||
echo "refusing: mooncake_master or sglang serve is still running" >&2
|
||||
pgrep -af '[m]ooncake_master|[s]glang serve' >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$SSD_DIR" ]; then
|
||||
echo "SSD cache dir does not exist: $SSD_DIR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "before: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
|
||||
find "$SSD_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
|
||||
echo "after: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
|
||||
df -h /data | tail -1
|
||||
@@ -0,0 +1,542 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
|
||||
\ 32 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
|
||||
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
|
||||
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,544 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
|
||||
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
|
||||
\ 16000000 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n\
|
||||
\ --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30\
|
||||
\ --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\
|
||||
\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,544 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
|
||||
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
|
||||
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
|
||||
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
|
||||
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
|
||||
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
|
||||
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
|
||||
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
|
||||
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
|
||||
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
|
||||
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
|
||||
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
|
||||
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
|
||||
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
|
||||
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
|
||||
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
|
||||
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
|
||||
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
|
||||
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
|
||||
\ 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
|
||||
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
|
||||
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
|
||||
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
|
||||
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
|
||||
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
|
||||
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
|
||||
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
|
||||
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
|
||||
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
|
||||
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
|
||||
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
|
||||
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
|
||||
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
|
||||
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
|
||||
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
|
||||
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
|
||||
\ 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
|
||||
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
|
||||
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
|
||||
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
|
||||
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
|
||||
\ 16000000 --g9i-p-qps 20 --g9i-p-refresh-ms 250 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
|
||||
\ 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
|
||||
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
|
||||
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
|
||||
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
|
||||
\ 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-l3ssd-test
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '549755813888'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- name: mc-0
|
||||
containerPort: 9400
|
||||
protocol: TCP
|
||||
- name: mc-1
|
||||
containerPort: 9401
|
||||
protocol: TCP
|
||||
- name: mc-2
|
||||
containerPort: 9402
|
||||
protocol: TCP
|
||||
- name: mc-3
|
||||
containerPort: 9403
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- name: gw-bin
|
||||
hostPath:
|
||||
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
|
||||
type: File
|
||||
@@ -0,0 +1,62 @@
|
||||
<h1>当前基线:ds41-gist-exp(4T L3 SSD,2026-11-17)</h1>
|
||||
<callout emoji="📌" background-color="light-blue"><p>本节是 2026-11-17 定稿的 c580/c620 基线,后续 Dynamo 叠加以此为底座。旧的 ds41-gist-1p1d 章节已归档,不再作为当前部署。</p></callout>
|
||||
|
||||
<h2>1. 基线身份与镜像</h2>
|
||||
<ul>
|
||||
<li><b>工作负载</b>:<code>ds41-gist-exp</code> / <code>cw-dblft6sgrhagmpdt8ef0</code>,namespace <code>prj-d7te67bj0sq2fgdrmbig</code>,b300-01。</li>
|
||||
<li><b>镜像</b>:<code>sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115</code>(40.1GB,含 Mooncake 0.3.14 + L3 SSD 支持)。</li>
|
||||
<li><b>manifest</b>:<code>deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml</code>;基线副本 <code>deploy/b300-k8s/baselines/20261117-dtok16-fastokens/manifest.yaml</code>;b300 副本在 <code>/data/ymk/deploy/b300-k8s/</code>。</li>
|
||||
<li><b>形态</b>:RBG 三角色 <code>p</code> / <code>d</code> / <code>gateway</code> + <code>mooncake-master</code>;全 hostNetwork / hostPID / privileged;P=GPU0-3,D=GPU4-7。</li>
|
||||
</ul>
|
||||
|
||||
<h2>2. 部署方式与端口</h2>
|
||||
<table>
|
||||
<thead><tr><th><p>角色</p></th><th><p>端口</p></th><th><p>说明</p></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><p>P</p></td><td><p>30000 / 8998 / 9400-9403</p></td><td><p>HTTP / bootstrap / Mooncake client metrics</p></td></tr>
|
||||
<tr><td><p>D</p></td><td><p>30300</p></td><td><p>HTTP</p></td></tr>
|
||||
<tr><td><p>gateway</p></td><td><p>30400 / 29000</p></td><td><p>业务 / metrics</p></td></tr>
|
||||
<tr><td><p>mooncake-master</p></td><td><p>50051 / 18081 / 19003</p></td><td><p>store / metadata / metrics</p></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>默认入口:<code>http://127.0.0.1:30400</code>;当前 new-api/AgentX 直接使用该入口。</p>
|
||||
|
||||
<h2>3. 关键参数</h2>
|
||||
<p><b>P(prefill)</b>:tp4 / dp4 / ep4,megamoe + flashmla_mega,ep-redundant 32,DSpark block 3,decoder SWA bounded replay;chunked-prefill 131072 / max-prefill 131072,max-running 512;fastokens + tokenizer-worker 16;HiCache ratio 4、page_first_direct、direct、write_through、Mooncake L3;<b>cache 模式,无 buffer_only、无 optimistic</b>;mem-fraction 0.80。</p>
|
||||
<p><b>D(decode)</b>:tp4 / dp4 / ep4,ep-redundant 64,DSpark block 3,stream-interval 32,swa-prefix-tails 2048;max-running 512,cuda-graph-max-bs-decode 512;chunked 32768 / max-prefill 8192;fastokens + tokenizer-worker 16;mem-fraction 0.80。</p>
|
||||
<p><b>gateway</b>:镜像内 <code>sgl-model-gateway</code>;<b>无 P admission cap</b>(无 p-qps / p-queue-reqs-cap / p-queue-token-cap,td-admit 默认 off);保留 health check + circuit breaker;<code>--g9i-decode-max-running 512</code>。这是 c580 能跑通的关键。</p>
|
||||
<p><b>L3</b>:Mooncake master 4TB SSD offload;<code>offload_on_evict=false</code>;L3 配置见 <code>/data/ymk/ds41/mooncake-store-l3ssd.json</code>。</p>
|
||||
|
||||
<h2>4. 启动 / 重启流程</h2>
|
||||
<pre lang="bash"><code># 1) 停完整工作负载并确认 GPU 0
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 "nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
|
||||
|
||||
# 2) 清 L3 SSD(实验/基线前必须)
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
|
||||
# 3) 用完整平台 manifest 启动(不要只 kubectl patch gateway)
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml
|
||||
|
||||
# 4) 等 4 Pod 1/1,检查 P/D/gateway health 200,再 smoke
|
||||
curl -sS http://127.0.0.1:30400/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"/data/models/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"1+1=?"}],"max_tokens":8}'
|
||||
</code></pre>
|
||||
<p>启动基线:P <code>max_total_num_tokens≈16,158,208</code> / 128 per-rank / avail ≈23GB;D <code>58,859,520</code> / 128 per-rank / avail ≈79GB。正常 warmup 6,385 条约 6.5 分钟。</p>
|
||||
|
||||
<h2>5. 基线结果</h2>
|
||||
<table>
|
||||
<thead><tr><th><p>场景</p></th><th><p>RPS</p></th><th><p>TTFT p50/p95/p99</p></th><th><p>ITL p95</p></th><th><p>结论</p></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><p>c580 基线</p></td><td><p>24.045</p></td><td><p>1.743 / 3.772 / 6.580 s</p></td><td><p>10.74 ms</p></td><td><p>推荐工作点</p></td></tr>
|
||||
<tr><td><p>c620 首轮(清 L3)</p></td><td><p>25.998</p></td><td><p>1.814 / 4.041 / 8.257 s</p></td><td><p>11.69 ms</p></td><td><p>吞吐 +8%,尾部变差</p></td></tr>
|
||||
<tr><td><p>c620 不重启 r2/r3</p></td><td><p>25.85 / 25.61</p></td><td><p>p95 5.39 / 6.44 s,p99 9.56 / 10.79 s</p></td><td><p>10.96 / 10.68 ms</p></td><td><p>D KV 接近/打满,TTFT 尾部持续劣化,不建议连测</p></td></tr>
|
||||
<tr><td><p>buffer_only</p></td><td><p>23.749</p></td><td><p>2.153 / 4.904 / 10.483 s</p></td><td><p>10.32 ms</p></td><td><p>吞吐 -1.2%、TTFT 更差,不采用</p></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>结论:<b>c580 是稳定基线</b>;c620 只适合短时冲吞吐,且每轮之间必须重启/清池。buffer_only 不作为基线。</p>
|
||||
|
||||
<h2>6. 作为 Dynamo 底座</h2>
|
||||
<p>当前镜像是 <b>SGLang + Mooncake 0.3.14 + L3 SSD</b> 基线,<b>不包含 Dynamo/nixl</b>。后续套 Dynamo 需要以本镜像为 base 另装 Dynamo 1.6.x / nixl,旧 <code>gist-nvlink-7f6fe2244-dynamo-local-20261026</code> 是更早的 10-23 代码,不是本基线。</p>
|
||||
<p>Dynamo 叠加时保留本页的 P/D 参数、L3 SSD 和 gateway 入口;先做单机 4P4D 冒烟,再评估 Dynamo 的 P/D 角色、discovery 和 KV 传输配置。</p>
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"master_server_address": "127.0.0.1:50051",
|
||||
"metadata_server": "P2PHANDSHAKE",
|
||||
"global_segment_size": "200gb",
|
||||
"protocol": "tcp",
|
||||
"enable_ssd_offload": true,
|
||||
"ssd_offload_path": "/data/ymk/mooncake-ssd"
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# L3 Mooncake SSD baseline requires the official 0.3.14 wheel.
|
||||
# Previous image shipped 0.3.13.post1; 0.3.14 contains the upstream SSD-offload fixes.
|
||||
# Wheel: https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
|
||||
# sha256: 74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
|
||||
# Installed with --no-deps; the base image already provides the runtime deps.
|
||||
mooncake-transfer-engine-cuda13==0.3.14
|
||||
@@ -0,0 +1,44 @@
|
||||
# 2026-11-16 buffer_only + optimistic 首启记录
|
||||
|
||||
## 变更(相对 `stable-20261116-pre-optimistic`)
|
||||
|
||||
P:
|
||||
|
||||
- 增加 `--hicache-host-memory-mode buffer_only`(保留 `--hicache-write-policy write_through`、
|
||||
`--hicache-ratio 4`、mooncake L3)
|
||||
- 增加 `--optimistic-prefill-attempts 4`
|
||||
|
||||
D:
|
||||
|
||||
- 增加 `--tokenizer-worker-num 16`(与 P 相同)
|
||||
|
||||
其余角色/参数不变;网关保持 live 的 `p-queue-reqs-cap 64 / token-cap 32M / p-qps 40`。
|
||||
|
||||
## 启动前清理
|
||||
|
||||
- `kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s` → `Terminated 0/0`,GPU 0-7 全 0 MiB
|
||||
- `/data/ymk/mooncake-ssd`:2.3T → 4.0K;`/data` 7.5T available
|
||||
- `kmctl update cw ds41-gist-exp -f ds41-gist-exp-20261116-bufferonly-optimistic.yaml`
|
||||
|
||||
## 首启验证(2026-11-16 22:00 CST)
|
||||
|
||||
- 4 个 Pod 全 `1/1 Running`;P/D/gateway `/health` 全 200
|
||||
- P `server_args`:`hicache_host_memory_mode='buffer_only'`、`optimistic_prefill_attempts=4`;
|
||||
没有 "Optimistic prefill supports ..." 的禁用 warning
|
||||
- D `server_args`:`tokenizer_worker_num=16`;日志有 16 个 multi-tokenizer worker
|
||||
- 启动基线:P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
|
||||
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`
|
||||
- 冒烟:chat `/v1/chat/completions` 200,回答正确;16 并发 4k 字符请求全 200
|
||||
- **optimistic 已生效**:P 日志出现 `#optimistic-req: 1`(其余为 0)
|
||||
- P/D 日志无 Traceback / ERROR / assert
|
||||
|
||||
## 回滚
|
||||
|
||||
```bash
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/sglang/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
|
||||
```
|
||||
|
||||
代码仓库标签:`stable-20261116-pre-optimistic`。
|
||||
@@ -0,0 +1,608 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T13:54:16Z"
|
||||
generation: 1
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30768532"
|
||||
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T14:01:52Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 1
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 1
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -0,0 +1,68 @@
|
||||
# 11/16 上线前稳定配置(ds41-gist-exp)
|
||||
|
||||
记录时间:2026-11-16 21:49 CST(13:49Z),在切换 `buffer_only + optimistic` 之前。
|
||||
|
||||
## 身份
|
||||
|
||||
- 平台 cw:`ds41-gist-exp` / `cw-dblft6sgrhagmpdt8ef0`
|
||||
- RBG:`ds41-gist-exp`(namespace `prj-d7te67bj0sq2fgdrmbig`)
|
||||
- 角色:`mooncake-master` / `p` / `d` / `gateway`
|
||||
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
|
||||
- P:GPU 0-3;D:GPU 4-7;全 hostNetwork / hostPID / privileged,钉 b300-01
|
||||
|
||||
## 文件
|
||||
|
||||
- `rbg-live-raw.yaml`:`kubectl get rbg -o yaml` 原始快照(含 status,仅存档)
|
||||
- `cw-ds41-gist-exp-11-16-pre-optimistic.yaml`:清理过平台字段的 RBG,可直接用于回滚更新
|
||||
- `cw-platform.yaml`:`kmctl get cw -o yaml` 的平台清单(`Spec.Manifest` 是旧值,见下方漂移)
|
||||
- 本说明
|
||||
|
||||
## 生效配置摘要
|
||||
|
||||
P:
|
||||
|
||||
- `tp4 / dp4 / ep4`、`megamoe`、`flashmla_mega`、`ep-redundant 32`
|
||||
- DSPARK block 3、decoder SWA bounded replay、`swa-full-tokens-ratio 0.12`
|
||||
- `chunked-prefill-size 131072`、`max-prefill-tokens 131072`
|
||||
- `max-running-requests 512`、`tokenizer-worker-num 16`
|
||||
- `hicache-ratio 4`、`hicache-mem-layout page_first_direct`、`hicache-io-backend direct`
|
||||
- `hicache-write-policy write_through`、`hicache-storage-backend mooncake`
|
||||
- **默认 host memory mode(cache),没有 `--hicache-host-memory-mode buffer_only`**
|
||||
- **没有 `--optimistic-prefill-attempts`**(`#optimistic-req` 为 0)
|
||||
- L3 SSD:4 TB `mooncake-ssd`,master `--offload_on_evict=false`
|
||||
|
||||
D:
|
||||
|
||||
- 同并行 tp4/dp4/ep4、`ep-redundant 64`、DSPARK block 3
|
||||
- `max-running-requests 512`、`cuda-graph-max-bs-decode 512`
|
||||
- `chunked-prefill-size 32768`、`max-prefill-tokens 8192`
|
||||
- **没有 `--tokenizer-worker-num`**(默认 1)
|
||||
- `mem-fraction-static 0.80`
|
||||
|
||||
Gateway:
|
||||
|
||||
- 实际 live:`--g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256`
|
||||
- `--g9i-regular-p-queue-reject-secs 0`、`--g9i-decode-max-running 512`
|
||||
- 网关二进制 hostPath:`/data/ymk/gw-ns-est-td/sgl-model-gateway`
|
||||
|
||||
> ⚠️ 平台 `cw-platform.yaml` 的 `Spec.Manifest` 里 gateway 仍是旧的
|
||||
> `p-queue-reqs-cap 32 / token-cap 16000000 / p-qps 20`;**live RBG 是 64 / 32M / 40**。
|
||||
> 本次快照与回滚文件都以 live RBG 为准,避免回滚时把网关参数打回旧值。
|
||||
|
||||
## 切换前状态
|
||||
|
||||
- 所有 Pod `1/1 Running`;P/D/gateway `/health` 200。
|
||||
- 为切 `buffer_only` 做 L3 SSD 清理前,`/data/ymk/mooncake-ssd` 占用 **2.3T**;清理后 4.0K,`/data` 7.5T available。
|
||||
- 启动基线(切换后复核):P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
|
||||
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`。
|
||||
|
||||
## 回滚
|
||||
|
||||
```bash
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
|
||||
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
|
||||
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
|
||||
-f /mnt/d/B300/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
|
||||
```
|
||||
|
||||
回滚前确认 `kmctl get cw ds41-gist-exp` 为 `Terminated 0/0`、GPU 0-7 为 0 MiB。
|
||||
+488
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,608 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T00:16:46Z"
|
||||
generation: 9
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30735810"
|
||||
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T12:36:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 9
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 9
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -0,0 +1,608 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T00:16:46Z"
|
||||
generation: 9
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "30735810"
|
||||
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 512 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-16T12:36:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 9
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 9
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -0,0 +1,13 @@
|
||||
# 2026-11-17 cache + gateway no-limit 实验参数(失败记录)
|
||||
|
||||
- P: hicache_host_memory_mode=cache, optimistic_prefill_attempts=0, tokenizer-worker-num=16,
|
||||
hicache ratio 4 / write_through / L3 mooncake, L3 SSD 4T.
|
||||
- D: tokenizer-worker-num=16.
|
||||
- Gateway: g9i-td-admit off, p-queue-reqs-cap 0, p-queue-token-cap 0, p-qps 0,
|
||||
decode-max-running 1000000, disable-circuit-breaker, disable-health-check.
|
||||
- Image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
- c580 run c580-cache-nolimit-20261117: warmup 4,720/6,385 (74%), 02:33:32 失败
|
||||
(ServerDisconnected/warmup_failure;P/D/gateway 0 restart,无 503)。
|
||||
- 观察:P 单批 32k/23k chunk 正常,但每 rank 同时仅 1 个请求;gateway
|
||||
smg_worker_requests_active 峰值 1,840,P queue 峰值 503,gateway 约 1 核。
|
||||
- 结论:该 no-limit/no-cb 网关配置不适合作为 c580 基线;仅作异常记录。
|
||||
@@ -0,0 +1,488 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '19003'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: '4'
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: '1'
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30000'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: '1'
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: '65536'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: '1'
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: '0'
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: '0'
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: '900'
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: '4'
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: '1'
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: '100'
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: '200'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.05'
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: '8'
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: '0'
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: '1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: '1.0'
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: '0'
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: '1'
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: '1'
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: '1'
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: '9400'
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: '1099511627776'
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: '0'
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '30300'
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: '1'
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: '1'
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: 'true'
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: 'true'
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: 'true'
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: '1'
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: '0.1'
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: '600'
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: '16'
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: '32'
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: '1024'
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '96'
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: '4'
|
||||
requests:
|
||||
cpu: '32'
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: '4'
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: '29000'
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: 'true'
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-type: custom-workload
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: '2'
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
@@ -0,0 +1,608 @@
|
||||
apiVersion: workloads.x-k8s.io/v1alpha2
|
||||
kind: RoleBasedGroup
|
||||
metadata:
|
||||
annotations:
|
||||
rbg.workloads.x-k8s.io/discovery-config-mode: refine
|
||||
creationTimestamp: "2026-11-16T13:54:16Z"
|
||||
generation: 6
|
||||
labels:
|
||||
app: ds41-gist-exp
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
sglang-model: ds-v41-flash
|
||||
name: ds41-gist-exp
|
||||
namespace: prj-d7te67bj0sq2fgdrmbig
|
||||
resourceVersion: "31040141"
|
||||
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
|
||||
spec:
|
||||
roles:
|
||||
- minReadySeconds: 0
|
||||
name: mooncake-master
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "19003"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-mooncake
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
|
||||
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
|
||||
exec /opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
|
||||
env:
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/master-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/master-xdg
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: mooncake-master
|
||||
ports:
|
||||
- containerPort: 19003
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
tcpSocket:
|
||||
port: 50051
|
||||
resources:
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: 64Gi
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: 8Gi
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: p
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30000"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-p
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--expert-balancedness-report-mode prometheus \
|
||||
--expert-distribution-recorder-buffer-size 64 \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
|
||||
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 0,1,2,3
|
||||
- name: SGLANG_LOG_MS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
|
||||
value: "1"
|
||||
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
|
||||
value: "65536"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DP_USE_GATHERV
|
||||
value: "1"
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
|
||||
value: "0"
|
||||
- name: SGLANG_NUMA_BIND_V2
|
||||
value: "0"
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: SGLANG_TIMEOUT_KEEP_ALIVE
|
||||
value: "900"
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
|
||||
value: /data/ymk/ds41/mooncake-store-l3ssd.json
|
||||
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
|
||||
value: "4"
|
||||
- name: SGLANG_HICACHE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
|
||||
value: "1"
|
||||
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
|
||||
value: "100"
|
||||
- name: SGLANG_MM_TRACE
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_MIN_MS
|
||||
value: "200"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.05"
|
||||
- name: SGLANG_BATCH_SPAN_MAX_LINKS
|
||||
value: "8"
|
||||
- name: SGLANG_BATCH_STAGE_TIMING
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_RECV_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_RECV_SPAN_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_MM_TRACE_SYNC
|
||||
value: "0"
|
||||
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
|
||||
value: "1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: SGLANG_TRACE_LOOP_SPAN
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_LOOP_SAMPLE
|
||||
value: "1.0"
|
||||
- name: SGLANG_FWD_TIMELINE
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_LOOP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_TRACE_MOONCAKE_DETAIL
|
||||
value: "0"
|
||||
- name: SGLANG_TRACE_BACKUP_DETAIL
|
||||
value: "1"
|
||||
- name: SGLANG_ASYNC_BACKUP_SUBMIT
|
||||
value: "1"
|
||||
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
|
||||
value: "1"
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
|
||||
value: "9400"
|
||||
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
|
||||
value: bucket_storage_backend
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
|
||||
value: "1099511627776"
|
||||
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
|
||||
value: fifo
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/p-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/p-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/p-hf
|
||||
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
|
||||
value: "0"
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
- containerPort: 8998
|
||||
name: bootstrap
|
||||
protocol: TCP
|
||||
- containerPort: 9400
|
||||
name: mc-0
|
||||
protocol: TCP
|
||||
- containerPort: 9401
|
||||
name: mc-1
|
||||
protocol: TCP
|
||||
- containerPort: 9402
|
||||
name: mc-2
|
||||
protocol: TCP
|
||||
- containerPort: 9403
|
||||
name: mc-3
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 2Ti
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 512Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- mooncake-master
|
||||
minReadySeconds: 0
|
||||
name: d
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "30300"
|
||||
prometheus.io/scheme: http
|
||||
labels:
|
||||
app: ds41-l3ssd-d
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec sglang serve \
|
||||
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
|
||||
--watchdog-timeout 3600
|
||||
env:
|
||||
- name: CUDA_VISIBLE_DEVICES
|
||||
value: 4,5,6,7
|
||||
- name: SGLANG_RAGGED_VERIFY_MODE
|
||||
value: static
|
||||
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
|
||||
value: "1"
|
||||
- name: SGLANG_DSV4_KV_LAYOUT
|
||||
value: v41_new
|
||||
- name: SGLANG_DEFAULT_THINKING
|
||||
value: "1"
|
||||
- name: SGLANG_DSV41_REASONING_EFFORT
|
||||
value: high
|
||||
- name: MC_INTRANODE_NVLINK
|
||||
value: "true"
|
||||
- name: MC_INTRA_NVLINK
|
||||
value: "true"
|
||||
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
|
||||
value: "true"
|
||||
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_TELEMETRY
|
||||
value: "1"
|
||||
- name: SGLANG_BATCH_SPAN_SAMPLE
|
||||
value: "0.1"
|
||||
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
|
||||
value: "600"
|
||||
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
|
||||
value: "16"
|
||||
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
|
||||
value: "32"
|
||||
- name: TZ
|
||||
value: Asia/Shanghai
|
||||
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
|
||||
value: "1024"
|
||||
- name: TMPDIR
|
||||
value: /data/ymk/l3ssd/d-tmp
|
||||
- name: XDG_CACHE_HOME
|
||||
value: /data/ymk/l3ssd/d-xdg
|
||||
- name: HF_HOME
|
||||
value: /data/ymk/l3ssd/d-hf
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: sglang
|
||||
ports:
|
||||
- containerPort: 30300
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 150
|
||||
httpGet:
|
||||
path: /health
|
||||
port: metrics
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "96"
|
||||
memory: 550Gi
|
||||
nvidia.com/gpu: "4"
|
||||
requests:
|
||||
cpu: "32"
|
||||
memory: 256Gi
|
||||
nvidia.com/gpu: "4"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /root/.cache/sglang
|
||||
name: sglang-cache
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /root/.cargo/bin/cargo
|
||||
name: fake-cargo
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/cache/sglang
|
||||
type: Directory
|
||||
name: sglang-cache
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 64Gi
|
||||
name: dshm
|
||||
- hostPath:
|
||||
path: /data/ymk/ds41/fake-cargo
|
||||
type: File
|
||||
name: fake-cargo
|
||||
- dependencies:
|
||||
- p
|
||||
- d
|
||||
minReadySeconds: 0
|
||||
name: gateway
|
||||
podManagementPolicy: Parallel
|
||||
replicas: 1
|
||||
standalonePattern:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "29000"
|
||||
prometheus.io/scheme: http
|
||||
prometheus.io/scrape: "true"
|
||||
labels:
|
||||
app: ds41-l3ssd-gw
|
||||
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
|
||||
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
|
||||
kongming.io/managed-by: kongming
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
|
||||
kongming.io/resource-type: custom-workload
|
||||
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
exec /opt/sglang/bin/sgl-model-gateway launch \
|
||||
--pd-disaggregation \
|
||||
--prefill http://127.0.0.1:30000 8998 \
|
||||
--decode http://127.0.0.1:30300 \
|
||||
--prefill-policy cache_aware --decode-policy cache_aware \
|
||||
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
|
||||
\
|
||||
\
|
||||
--g9i-regular-p-queue-reject-secs 0 \
|
||||
--g9i-decode-max-running 1000000 \
|
||||
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
|
||||
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
|
||||
--cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \
|
||||
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
|
||||
--host 0.0.0.0 --port 30400
|
||||
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: gateway
|
||||
ports:
|
||||
- containerPort: 30400
|
||||
name: http
|
||||
protocol: TCP
|
||||
- containerPort: 29000
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
failureThreshold: 60
|
||||
httpGet:
|
||||
path: /health
|
||||
port: http
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 32Gi
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- mountPath: /data
|
||||
name: data
|
||||
- mountPath: /opt/sglang/bin/sgl-model-gateway
|
||||
name: gw-bin
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostNetwork: true
|
||||
nodeSelector:
|
||||
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
|
||||
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
|
||||
kubernetes.io/hostname: b300-01
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /data
|
||||
type: Directory
|
||||
name: data
|
||||
- hostPath:
|
||||
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
|
||||
type: File
|
||||
name: gw-bin
|
||||
status:
|
||||
conditions:
|
||||
- lastTransitionTime: "2026-11-17T01:58:12Z"
|
||||
message: All roles are ready
|
||||
observedGeneration: 6
|
||||
reason: AllRolesReady
|
||||
status: "True"
|
||||
type: Ready
|
||||
observedGeneration: 6
|
||||
roleStatuses:
|
||||
- name: mooncake-master
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: p
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: d
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
- name: gateway
|
||||
readyReplicas: 1
|
||||
replicas: 1
|
||||
updatedReplicas: 1
|
||||
@@ -56,6 +56,7 @@ class AnthropicUsage(BaseModel):
|
||||
class TextBlock(BaseModel):
|
||||
type: Literal["text"] = "text"
|
||||
text: str
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class ImageBlock(BaseModel):
|
||||
@@ -91,6 +92,50 @@ class ToolReferenceBlock(BaseModel):
|
||||
id: Optional[str] = None
|
||||
|
||||
|
||||
class McpToolReference(BaseModel):
|
||||
type: Literal["mcp_tool_reference"]
|
||||
server_name: str
|
||||
name: str
|
||||
|
||||
|
||||
class McpToolsetReference(BaseModel):
|
||||
type: Literal["mcp_toolset_reference"]
|
||||
server_name: str
|
||||
|
||||
|
||||
class ToolDefinition(BaseModel):
|
||||
"""A tool given by value: `definition` is a `tools` entry."""
|
||||
|
||||
type: Literal["tool_definition"]
|
||||
definition: dict[str, Any]
|
||||
|
||||
|
||||
class ToolAdditionBlock(BaseModel):
|
||||
"""Anthropic beta block: a ``system`` message offers a tool from its position
|
||||
on. Rendered through deepseek-recipe, which reads only a ``tool_reference``
|
||||
target."""
|
||||
|
||||
type: Literal["tool_addition"] = "tool_addition"
|
||||
tool: Annotated[
|
||||
Union[
|
||||
ToolReferenceBlock, McpToolReference, McpToolsetReference, ToolDefinition
|
||||
],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class ToolRemovalBlock(BaseModel):
|
||||
"""Anthropic beta block: a ``system`` message withdraws a tool."""
|
||||
|
||||
type: Literal["tool_removal"] = "tool_removal"
|
||||
tool: Annotated[
|
||||
Union[ToolReferenceBlock, McpToolReference, McpToolsetReference],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
cache_control: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
class SearchResultBlock(BaseModel):
|
||||
type: Literal["search_result"] = "search_result"
|
||||
# ``source`` here is a URL/identifier string (unlike ImageBlock.source).
|
||||
@@ -117,6 +162,8 @@ AnthropicContentBlock = Annotated[
|
||||
ToolUseBlock,
|
||||
ToolResultBlock,
|
||||
ToolReferenceBlock,
|
||||
ToolAdditionBlock,
|
||||
ToolRemovalBlock,
|
||||
SearchResultBlock,
|
||||
ThinkingBlock,
|
||||
RedactedThinkingBlock,
|
||||
@@ -128,6 +175,8 @@ AnthropicContentBlock = Annotated[
|
||||
class AnthropicMessage(BaseModel):
|
||||
role: Literal["user", "assistant", "system"]
|
||||
content: Union[str, list[AnthropicContentBlock]]
|
||||
clear_at: Optional[Literal["next_user_message", "never"]] = None
|
||||
output_config: Optional[dict[str, Any]] = None
|
||||
|
||||
|
||||
# ---------- Tools (discriminated by ``type`` family) ----------
|
||||
|
||||
@@ -20,8 +20,10 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
AnthropicContentBlock,
|
||||
AnthropicCountTokensRequest,
|
||||
AnthropicCountTokensResponse,
|
||||
AnthropicCustomTool,
|
||||
AnthropicError,
|
||||
AnthropicErrorResponse,
|
||||
AnthropicMessage,
|
||||
AnthropicMessageEndDelta,
|
||||
AnthropicMessagesRequest,
|
||||
AnthropicMessagesResponse,
|
||||
@@ -32,6 +34,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
ContentBlockStopEvent,
|
||||
ErrorEvent,
|
||||
InputJsonDelta,
|
||||
McpToolReference,
|
||||
McpToolsetReference,
|
||||
MessageDeltaEvent,
|
||||
MessageStartEvent,
|
||||
MessageStopEvent,
|
||||
@@ -40,6 +44,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
|
||||
TextDelta,
|
||||
ThinkingBlock,
|
||||
ThinkingDelta,
|
||||
ToolDefinition,
|
||||
ToolReferenceBlock,
|
||||
ToolUseBlock,
|
||||
is_server_tool,
|
||||
)
|
||||
@@ -152,6 +158,207 @@ def _extract_system_text(
|
||||
return "\n".join(texts) if texts else None
|
||||
|
||||
|
||||
def _block_types(request: AnthropicMessagesRequest) -> set[str]:
|
||||
"""The block types of the request's system and messages, tool result contents
|
||||
included."""
|
||||
types = set()
|
||||
for content in [request.system, *(message.content for message in request.messages)]:
|
||||
for block in content if isinstance(content, list) else []:
|
||||
types.add(block.type)
|
||||
if block.type == "tool_result" and isinstance(block.content, list):
|
||||
types.update(item.type for item in block.content)
|
||||
return types
|
||||
|
||||
|
||||
def _without_cleared_messages(
|
||||
request: AnthropicMessagesRequest,
|
||||
) -> AnthropicMessagesRequest:
|
||||
"""`request` without the turn-scoped system messages (`clear_at:
|
||||
"next_user_message"`) that a later user message has cleared."""
|
||||
last_user = max(
|
||||
(
|
||||
index
|
||||
for index, message in enumerate(request.messages)
|
||||
if message.role == "user"
|
||||
),
|
||||
default=-1,
|
||||
)
|
||||
rendered = []
|
||||
for index, message in enumerate(request.messages):
|
||||
if message.clear_at is not None and message.role != "system":
|
||||
raise ValueError(
|
||||
f"messages.{index}.clear_at: clear_at is only permitted on role "
|
||||
"'system' messages"
|
||||
)
|
||||
if message.clear_at == "next_user_message":
|
||||
_check_turn_scoped(index=index, message=message)
|
||||
_check_turn_scoped_placement(index=index, messages=request.messages)
|
||||
if index < last_user:
|
||||
continue
|
||||
rendered.append(message)
|
||||
return request.model_copy(update={"messages": rendered})
|
||||
|
||||
|
||||
def _check_turn_scoped(index: int, message: AnthropicMessage) -> None:
|
||||
scope = "a turn-scoped system message (clear_at: 'next_user_message')"
|
||||
if message.output_config is not None:
|
||||
raise ValueError(f"messages.{index}: output_config is not permitted on {scope}")
|
||||
for position, block in enumerate(
|
||||
message.content if isinstance(message.content, list) else []
|
||||
):
|
||||
if block.type != "text":
|
||||
raise ValueError(
|
||||
f"messages.{index}: a turn-scoped system message supports text "
|
||||
"blocks only (clear_at: 'next_user_message')"
|
||||
)
|
||||
if block.cache_control is not None:
|
||||
raise ValueError(
|
||||
f"messages.{index}.content.{position}: cache_control is not "
|
||||
f"permitted on {scope}"
|
||||
)
|
||||
|
||||
|
||||
def _check_turn_scoped_placement(index: int, messages: list[AnthropicMessage]) -> None:
|
||||
# Consecutive system messages are placed as one group.
|
||||
before = next(
|
||||
(m.role for m in reversed(messages[:index]) if m.role != "system"), None
|
||||
)
|
||||
after = next((m.role for m in messages[index + 1 :] if m.role != "system"), None)
|
||||
if before != "user" or after == "user":
|
||||
raise ValueError(
|
||||
f"messages.{index}: a turn-scoped system message (clear_at: "
|
||||
"'next_user_message') must follow a user message and either end "
|
||||
"messages or precede an assistant message"
|
||||
)
|
||||
|
||||
|
||||
def _declare_inline_tools(
|
||||
request: AnthropicMessagesRequest,
|
||||
) -> AnthropicMessagesRequest:
|
||||
"""`request` with each custom tool a `tool_addition` block defines by value
|
||||
declared in `tools` as deferred, and the block naming it by reference. A
|
||||
definition of a tool already offered unchanged is dropped."""
|
||||
tools = list(request.tools or [])
|
||||
declared = {tool.name: tool for tool in tools}
|
||||
offered = {tool.name for tool in tools if not tool.defer_loading}
|
||||
unresolved: dict[str, str] = {}
|
||||
messages = []
|
||||
for index, message in enumerate(request.messages):
|
||||
if isinstance(message.content, str):
|
||||
messages.append(message)
|
||||
continue
|
||||
blocks = []
|
||||
for position, block in enumerate(message.content):
|
||||
where = f"messages.{index}.content.{position}"
|
||||
for name in _referenced_tool_names(block):
|
||||
if name not in declared:
|
||||
unresolved.setdefault(name, where)
|
||||
if block.type == "tool_addition" and block.tool.type != "tool_reference":
|
||||
if message.role != "system":
|
||||
raise ValueError(
|
||||
f"{where}: tool_addition blocks can only be in system messages"
|
||||
)
|
||||
tool = _inline_custom_tool(where=where, target=block.tool)
|
||||
# Declaring the tool makes it resolvable from the first message on.
|
||||
if tool.name in unresolved:
|
||||
raise ValueError(
|
||||
f"{unresolved[tool.name]}: tool reference {tool.name!r} "
|
||||
f"precedes the tool_definition that defines it at {where}"
|
||||
)
|
||||
earlier = declared.setdefault(tool.name, tool)
|
||||
if earlier is tool:
|
||||
tools.append(tool)
|
||||
elif is_server_tool(earlier) or _definition(earlier) != _definition(
|
||||
tool
|
||||
):
|
||||
raise ValueError(
|
||||
f"{where}: tool_definition "
|
||||
f"reuses the name {tool.name!r} with a different "
|
||||
"definition; replacing a tool mid-conversation is not "
|
||||
"supported"
|
||||
)
|
||||
if tool.name in offered:
|
||||
continue
|
||||
offered.add(tool.name)
|
||||
block = block.model_copy(
|
||||
update={"tool": ToolReferenceBlock(name=tool.name)}
|
||||
)
|
||||
blocks.append(block)
|
||||
# recipe renders a system message left without blocks as an empty turn.
|
||||
if blocks or not message.content:
|
||||
messages.append(message.model_copy(update={"content": blocks}))
|
||||
return request.model_copy(update={"tools": tools, "messages": messages})
|
||||
|
||||
|
||||
def _referenced_tool_names(block: AnthropicContentBlock) -> list[Optional[str]]:
|
||||
if block.type == "tool_addition" and block.tool.type == "tool_reference":
|
||||
return [block.tool.name or block.tool.tool_name]
|
||||
if block.type == "tool_reference":
|
||||
return [block.tool_name or block.name]
|
||||
if block.type == "tool_result" and isinstance(block.content, list):
|
||||
return [name for item in block.content for name in _referenced_tool_names(item)]
|
||||
return []
|
||||
|
||||
|
||||
def _inline_custom_tool(
|
||||
where: str, target: Union[McpToolReference, McpToolsetReference, ToolDefinition]
|
||||
) -> AnthropicCustomTool:
|
||||
if target.type != "tool_definition":
|
||||
raise ValueError(
|
||||
f"{where}: tool_addition with a {target.type} is not supported; this "
|
||||
"server runs no MCP connector"
|
||||
)
|
||||
kind = target.definition.get("type")
|
||||
if kind not in (None, "custom"):
|
||||
raise ValueError(
|
||||
f"{where}: tool_definition supports custom tools only, got a "
|
||||
f"definition of type {kind!r}"
|
||||
)
|
||||
tool = AnthropicCustomTool.model_validate(target.definition)
|
||||
if tool.defer_loading:
|
||||
raise ValueError(
|
||||
f"{where}: a tool_definition with defer_loading is not supported; "
|
||||
"declare the tool in tools and add it by reference"
|
||||
)
|
||||
return tool.model_copy(update={"defer_loading": True})
|
||||
|
||||
|
||||
def _definition(tool: AnthropicCustomTool) -> dict[str, Any]:
|
||||
return tool.model_dump(exclude={"type", "defer_loading"})
|
||||
|
||||
|
||||
def _native_block(block: dict[str, Any]) -> dict[str, Any]:
|
||||
"""`block` with SGLang's accepted aliases spelled as deepseek-recipe reads them."""
|
||||
kind = block["type"]
|
||||
if kind == "tool_result":
|
||||
content = block.get("content")
|
||||
fields = {
|
||||
"type": kind,
|
||||
"tool_use_id": block.get("tool_use_id") or block.get("id"),
|
||||
"content": (
|
||||
[_native_block(item) for item in content]
|
||||
if isinstance(content, list)
|
||||
else content
|
||||
),
|
||||
"is_error": block.get("is_error"),
|
||||
}
|
||||
return {key: value for key, value in fields.items() if value is not None}
|
||||
if kind == "tool_reference":
|
||||
return {"type": kind, "tool_name": block.get("tool_name") or block.get("name")}
|
||||
if kind == "tool_addition":
|
||||
tool = block["tool"]
|
||||
name = tool.get("name") or tool.get("tool_name")
|
||||
return {"type": kind, "tool": {"type": "tool_reference", "name": name}}
|
||||
return block
|
||||
|
||||
|
||||
def _native_message(message: AnthropicMessage) -> dict[str, Any]:
|
||||
data = message.model_dump(exclude_none=True)
|
||||
if isinstance(data["content"], list):
|
||||
data["content"] = [_native_block(block) for block in data["content"]]
|
||||
return data
|
||||
|
||||
|
||||
def _wrap_sse_event(data: str, event_type: str) -> str:
|
||||
"""Format an Anthropic SSE event with event type and data lines."""
|
||||
return f"event: {event_type}\ndata: {data}\n\n"
|
||||
@@ -218,6 +425,24 @@ class AnthropicServing:
|
||||
self, anthropic_request: AnthropicMessagesRequest
|
||||
) -> ChatCompletionRequest:
|
||||
"""Convert an Anthropic Messages request to an OpenAI ChatCompletion request."""
|
||||
anthropic_request = _declare_inline_tools(
|
||||
_without_cleared_messages(anthropic_request)
|
||||
)
|
||||
block_types = _block_types(anthropic_request)
|
||||
if "tool_removal" in block_types:
|
||||
raise ValueError(
|
||||
"tool_removal blocks are not supported; a tool cannot be withdrawn "
|
||||
"mid-conversation"
|
||||
)
|
||||
native = self.openai_serving_chat.native_tool_loading and (
|
||||
bool(block_types & {"tool_reference", "tool_addition"})
|
||||
or any(
|
||||
not is_server_tool(tool) and tool.defer_loading
|
||||
for tool in anthropic_request.tools or []
|
||||
)
|
||||
)
|
||||
if not native and "tool_addition" in block_types:
|
||||
raise ValueError("tool_addition blocks are not supported by this model")
|
||||
openai_messages = []
|
||||
|
||||
def _convert_anthropic_image_source_to_openai_part(
|
||||
@@ -719,8 +944,51 @@ class AnthropicServing:
|
||||
elif chat_request.tools:
|
||||
chat_request.tool_choice = "auto"
|
||||
|
||||
if native:
|
||||
self._prepare_native_prompt(anthropic_request, chat_request)
|
||||
return chat_request
|
||||
|
||||
def _prepare_native_prompt(
|
||||
self,
|
||||
anthropic_request: AnthropicMessagesRequest,
|
||||
chat_request: ChatCompletionRequest,
|
||||
) -> None:
|
||||
"""Render the request's own wire form into `chat_request`, whose tools
|
||||
become the ones the history has loaded."""
|
||||
body = {
|
||||
"messages": [
|
||||
_native_message(message) for message in anthropic_request.messages
|
||||
],
|
||||
# recipe reads a tool with a `type` as a server tool.
|
||||
"tools": [
|
||||
tool.model_dump(exclude_none=True, exclude={"type"})
|
||||
for tool in anthropic_request.tools or []
|
||||
if not is_server_tool(tool)
|
||||
],
|
||||
}
|
||||
system = anthropic_request.system
|
||||
if system:
|
||||
body["system"] = (
|
||||
system
|
||||
if isinstance(system, str)
|
||||
else [block.model_dump(exclude_none=True) for block in system]
|
||||
)
|
||||
choice = anthropic_request.tool_choice
|
||||
if choice is not None and choice.type == "none":
|
||||
body["tool_choice"] = {"type": "none"}
|
||||
tools = self.openai_serving_chat.prepare_native_prompt(
|
||||
chat_request, protocol="messages", body=body
|
||||
).tools
|
||||
if (
|
||||
choice is not None
|
||||
and choice.type == "tool"
|
||||
and not any(tool.function.name == choice.name for tool in tools)
|
||||
):
|
||||
raise ValueError(
|
||||
f"tool_choice names {choice.name!r}, a deferred tool no "
|
||||
"tool_reference or tool_addition block has loaded"
|
||||
)
|
||||
|
||||
async def _handle_non_streaming(
|
||||
self,
|
||||
chat_request: ChatCompletionRequest,
|
||||
|
||||
@@ -3,7 +3,7 @@ concrete adapters load only when selected."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol
|
||||
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol, Union
|
||||
|
||||
import msgspec
|
||||
|
||||
@@ -15,8 +15,27 @@ if TYPE_CHECKING:
|
||||
ToolCall,
|
||||
)
|
||||
from sglang.srt.function_call.core_types import ToolCallItem
|
||||
from sglang.srt.multimodal.media_io import ImageData
|
||||
|
||||
EncodingBackend = Literal["builtin", "recipe"]
|
||||
HistoryToolCallArguments = Literal["strict", "lenient", "as_sent"]
|
||||
NativeProtocol = Literal["responses", "messages"]
|
||||
|
||||
|
||||
class NativePrompt(msgspec.Struct, frozen=True, kw_only=True):
|
||||
"""A Responses or Messages request rendered from its own wire form."""
|
||||
|
||||
prompt_ids: list[int]
|
||||
tools: list[Tool]
|
||||
"""The tools the model can call after the request's history: those loaded from
|
||||
the start and those the history has loaded since."""
|
||||
image_data: list[ImageData] = []
|
||||
"""The images the prompt's placeholders stand for, in order."""
|
||||
namespace_names: dict[str, tuple[str, str]] = {}
|
||||
"""The call name of each namespace member, mapped to its namespace and the
|
||||
name the request gives it there."""
|
||||
continued_turn: Optional[dict[str, Any]] = None
|
||||
"""The final assistant message the prompt continues, as a chat message."""
|
||||
|
||||
|
||||
class ChatOutputHooks(msgspec.Struct, frozen=True, kw_only=True):
|
||||
@@ -78,6 +97,25 @@ class ChatEncodingOutputAdapter(Protocol):
|
||||
) -> ResponsesOutputStream: ...
|
||||
|
||||
|
||||
class NativeRequestEncoder(Protocol):
|
||||
def encode_native(
|
||||
self,
|
||||
*,
|
||||
protocol: NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
thinking: bool,
|
||||
effort: Union[str, int, float, None],
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> NativePrompt:
|
||||
"""``body`` carries the request's conversation, tools and, for
|
||||
``tool_choice="none"``, its choice; ``thinking`` and ``effort`` are the
|
||||
handler's decisions, as for ``encode_messages``. ``response_schema`` is
|
||||
the JSON schema the answer follows; a trailing assistant turn is continued
|
||||
under ``continue_final_message`` and otherwise re-sent as a user turn."""
|
||||
...
|
||||
|
||||
|
||||
class ChatEncodingAdapter(Protocol):
|
||||
collects_openai_media: bool
|
||||
"""Whether media still has to be collected from the OpenAI messages after
|
||||
@@ -92,6 +130,12 @@ class ChatEncodingAdapter(Protocol):
|
||||
"""None when the configured parsers leave output to the text parsers."""
|
||||
...
|
||||
|
||||
@property
|
||||
def native_encoder(self) -> Optional[NativeRequestEncoder]:
|
||||
"""None when Responses and Messages requests that load tools mid-conversation
|
||||
have no rendering of their own wire form."""
|
||||
...
|
||||
|
||||
def encode_messages(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -17,6 +17,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
import msgspec
|
||||
import unicodedata
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
@@ -743,6 +744,10 @@ class ChatCompletionMessageGenericParam(BaseModel):
|
||||
name: Optional[str] = None
|
||||
phase: Optional[Literal["commentary", "final_answer"]] = None
|
||||
reasoning_content: Optional[str] = None
|
||||
action: Optional[str] = Field(
|
||||
default=None,
|
||||
description="Action tag for assistant messages (used by deepseek-recipe).",
|
||||
)
|
||||
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
|
||||
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
|
||||
# DeepSeek API prefix completion: continue this final assistant message.
|
||||
@@ -796,14 +801,22 @@ class Function(BaseModel):
|
||||
description: Optional[str] = Field(default=None, examples=[None])
|
||||
name: str
|
||||
parameters: Optional[object] = None
|
||||
strict: bool = False
|
||||
strict: Optional[bool] = None
|
||||
defer_loading: Optional[bool] = None
|
||||
namespace: Optional[str] = Field(
|
||||
default=None,
|
||||
description="Tool namespace for disambiguation (used by deepseek-recipe).",
|
||||
)
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
def _serialize(self, handler):
|
||||
data = handler(self)
|
||||
if self.defer_loading is None:
|
||||
data.pop("defer_loading", None)
|
||||
if self.strict is None:
|
||||
data.pop("strict", None)
|
||||
if self.namespace is None:
|
||||
data.pop("namespace", None)
|
||||
return data
|
||||
|
||||
|
||||
@@ -821,6 +834,26 @@ class Tool(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class ChatCompletionNamespaceTool(BaseModel):
|
||||
"""A Responses-style namespace grouping sent to the Chat Completions API.
|
||||
|
||||
Chat Completions has no namespace field in its tool-call shape, so the
|
||||
members are flattened to ``<namespace>__<function>`` before rendering; the
|
||||
model calls that flat name and the response reports it back unchanged.
|
||||
"""
|
||||
|
||||
type: Literal["namespace"]
|
||||
name: str
|
||||
description: Optional[str] = None
|
||||
tools: List[Tool]
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_members(self) -> "ChatCompletionNamespaceTool":
|
||||
if not self.tools:
|
||||
raise ValueError("namespace tools must list at least one function tool")
|
||||
return self
|
||||
|
||||
|
||||
# Tool is defined after the message params that reference it, so the forward
|
||||
# reference has to be resolved explicitly.
|
||||
ChatCompletionMessageGenericParam.model_rebuild()
|
||||
@@ -908,11 +941,22 @@ class ChatCompletionRequest(PDRoutingFields):
|
||||
temperature: Optional[float] = None
|
||||
top_p: Optional[float] = None
|
||||
user: Optional[str] = None
|
||||
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
|
||||
tools: Optional[List[Union[Tool, ChatCompletionNamespaceTool]]] = Field(
|
||||
default=None, examples=[None]
|
||||
)
|
||||
# Set by /v1/responses when the request has namespace tools: the chat template
|
||||
# renders these, while ``tools`` (flattened to ``<namespace>.<function>``) still
|
||||
# drives tool-call parsing.
|
||||
_template_tools: Optional[List[Dict[str, Any]]] = PrivateAttr(default=None)
|
||||
# Set with ``input_ids`` by a caller that rendered them itself: the images the
|
||||
# prompt's placeholders stand for, in order.
|
||||
_input_image_data: Optional[List[Any]] = PrivateAttr(default=None)
|
||||
# Whether ``tools`` are the only tools the rendered prompt shows, so a call
|
||||
# to any other tool is dropped rather than returned.
|
||||
_drops_unlisted_tool_calls: bool = PrivateAttr(default=False)
|
||||
# Set with ``input_ids`` under ``continue_final_message``: the final assistant
|
||||
# message the prompt continues, where output parsing resumes.
|
||||
_continued_turn: Optional[Dict[str, Any]] = PrivateAttr(default=None)
|
||||
tool_choice: Union[ToolChoice, Literal["auto", "required", "none"]] = Field(
|
||||
default="auto", examples=["none"]
|
||||
) # noqa
|
||||
@@ -1140,6 +1184,39 @@ class ChatCompletionRequest(PDRoutingFields):
|
||||
|
||||
return values
|
||||
|
||||
@model_validator(mode="after")
|
||||
def flatten_namespace_tools(self) -> "ChatCompletionRequest":
|
||||
"""Render namespace groups as flat ``ns__function`` chat tools."""
|
||||
if not self.tools:
|
||||
return self
|
||||
flattened: List[Tool] = []
|
||||
for tool in self.tools:
|
||||
if isinstance(tool, ChatCompletionNamespaceTool):
|
||||
for member in tool.tools:
|
||||
name = f"{tool.name}__{member.function.name}"
|
||||
description = member.function.description
|
||||
if tool.description:
|
||||
description = (
|
||||
tool.description + chr(10) + description
|
||||
if description
|
||||
else tool.description
|
||||
)
|
||||
flattened.append(
|
||||
Tool(
|
||||
type="function",
|
||||
function=Function(
|
||||
name=name,
|
||||
description=description,
|
||||
parameters=member.function.parameters,
|
||||
strict=member.function.strict,
|
||||
),
|
||||
)
|
||||
)
|
||||
else:
|
||||
flattened.append(tool)
|
||||
self.tools = flattened
|
||||
return self
|
||||
|
||||
def to_sampling_params(
|
||||
self,
|
||||
stop: List[str],
|
||||
@@ -1252,6 +1329,7 @@ class ChatMessage(BaseModel):
|
||||
role: Optional[str] = None
|
||||
content: Optional[str] = None
|
||||
reasoning_content: Optional[str] = None
|
||||
action: Optional[str] = None
|
||||
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
@@ -1259,6 +1337,8 @@ class ChatMessage(BaseModel):
|
||||
data = handler(self)
|
||||
if self.tool_calls is None:
|
||||
data.pop("tool_calls", None)
|
||||
if self.action is None:
|
||||
data.pop("action", None)
|
||||
return data
|
||||
|
||||
|
||||
@@ -1811,6 +1891,8 @@ RESPONSE_TOOL_TYPES = Literal[
|
||||
class ResponseTool(BaseModel):
|
||||
"""Tool definition for responses."""
|
||||
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
type: RESPONSE_TOOL_TYPES = Field(description="Type of tool to enable")
|
||||
name: Optional[str] = None
|
||||
description: Optional[str] = None
|
||||
@@ -1830,6 +1912,40 @@ class ResponseTool(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class ResponseToolState(msgspec.Struct, frozen=True):
|
||||
"""Public tool metadata and name mappings for one translated request."""
|
||||
|
||||
public_tools: list[ResponseTool] | None = None
|
||||
public_choice: str | dict[str, Any] | None = None
|
||||
names: dict[str, dict[str, str]] = {}
|
||||
# The tools the model can call after the history, from a request rendered in
|
||||
# its own wire form; None when `tools` alone declares them.
|
||||
chat_tools: list[Tool] | None = None
|
||||
# ``custom`` tools a ``tool_search_output`` loaded, beside the declared ones.
|
||||
discovered_custom_names: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
class ResponseToolSearchCall(BaseModel):
|
||||
"""A client-executed ``tool_search`` call; the pinned openai SDK has no type."""
|
||||
|
||||
id: str
|
||||
type: Literal["tool_search_call"] = "tool_search_call"
|
||||
call_id: str
|
||||
execution: Literal["client"] = "client"
|
||||
arguments: Dict[str, Any]
|
||||
status: Literal["in_progress", "completed", "incomplete"]
|
||||
|
||||
|
||||
class ResponseToolSearchCallEvent(BaseModel):
|
||||
"""``response.output_item.added`` / ``.done`` for a tool search call, which the
|
||||
SDK's event types reject."""
|
||||
|
||||
type: Literal["response.output_item.added", "response.output_item.done"]
|
||||
sequence_number: int
|
||||
output_index: int
|
||||
item: ResponseToolSearchCall
|
||||
|
||||
|
||||
class ResponseInputMessageParam(EasyInputMessageParam, total=False):
|
||||
phase: Optional[Literal["commentary", "final_answer"]]
|
||||
|
||||
@@ -1856,6 +1972,10 @@ ResponseInputOutputItem: TypeAlias = Union[
|
||||
class ResponsesRequest(PDRoutingFields):
|
||||
"""Request body for v1/responses endpoint."""
|
||||
|
||||
_tool_state: ResponseToolState = PrivateAttr(default_factory=ResponseToolState)
|
||||
# By call_id, the name the model called a tool by when the output item
|
||||
# reports another; the stored history replays the call under it.
|
||||
_called_tool_names: Dict[str, str] = PrivateAttr(default_factory=dict)
|
||||
_response_parser_prefix: str = PrivateAttr(default="")
|
||||
_response_parser_grammar_start: Optional[str] = PrivateAttr(default=None)
|
||||
_response_parser_json_response_format: bool = PrivateAttr(default=False)
|
||||
@@ -2052,7 +2172,10 @@ class ResponsesRequest(PDRoutingFields):
|
||||
"name"
|
||||
)
|
||||
if tool_choice.get("type") in ("function", "custom") and name:
|
||||
return {"type": "function", "name": name}
|
||||
choice = {"type": tool_choice["type"], "name": name}
|
||||
if "namespace" in tool_choice:
|
||||
choice["namespace"] = tool_choice["namespace"]
|
||||
return choice
|
||||
return "auto"
|
||||
|
||||
def to_sampling_params(
|
||||
@@ -2176,6 +2299,7 @@ class ResponsesResponse(BaseModel):
|
||||
ResponseOutputItem,
|
||||
ResponseReasoningItem,
|
||||
ResponseFunctionToolCall,
|
||||
ResponseToolSearchCall,
|
||||
]
|
||||
] = Field(default_factory=list)
|
||||
status: Literal[
|
||||
@@ -2241,6 +2365,7 @@ class ResponsesResponse(BaseModel):
|
||||
ResponseOutputItem,
|
||||
ResponseReasoningItem,
|
||||
ResponseFunctionToolCall,
|
||||
ResponseToolSearchCall,
|
||||
]
|
||||
],
|
||||
status: str,
|
||||
|
||||
@@ -957,12 +957,13 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
The reasoning parser is told the same thing, so channel-framed formats
|
||||
keep their framing intact exactly when a tool-call parser consumes it.
|
||||
"""
|
||||
return bool(
|
||||
request.tool_choice != "none"
|
||||
and self._effective_tools(request)
|
||||
and self.tool_call_parser
|
||||
and not request._response_parser_json_response_format
|
||||
)
|
||||
if not self.tool_call_parser or request._response_parser_json_response_format:
|
||||
return False
|
||||
if request._drops_unlisted_tool_calls and request.tool_choice == "none":
|
||||
# The history showed the model tools, so it can still write a call;
|
||||
# parsing drops it, where skipping the parser would return its markup.
|
||||
return True
|
||||
return bool(request.tool_choice != "none" and self._effective_tools(request))
|
||||
|
||||
def _validate_request(self, request: ChatCompletionRequest) -> str | None:
|
||||
"""Validate that the input is valid."""
|
||||
@@ -1367,18 +1368,7 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
self, request: ChatCompletionRequest, is_multimodal: bool
|
||||
) -> MessageProcessingResult:
|
||||
"""Process chat messages and apply chat template"""
|
||||
if self.default_chat_template_kwargs:
|
||||
ctk = dict(request.chat_template_kwargs or {})
|
||||
for k, v in self.default_chat_template_kwargs.items():
|
||||
ctk.setdefault(k, v)
|
||||
request.chat_template_kwargs = ctk
|
||||
effort = ctk.get("reasoning_effort")
|
||||
if effort is not None and request.reasoning_effort is None:
|
||||
request.reasoning_effort = effort
|
||||
|
||||
normalize_hunyuan_reasoning_effort(
|
||||
request, self.reasoning_parser, self.template_manager.reasoning_config
|
||||
)
|
||||
self._apply_request_defaults(request)
|
||||
|
||||
# GptOss model needs to keep special tokens for harmony parsing
|
||||
if self.is_gpt_oss or self.is_gemma4:
|
||||
@@ -1484,7 +1474,7 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
result = MessageProcessingResult(
|
||||
prompt="",
|
||||
prompt_ids=request.input_ids,
|
||||
image_data=None,
|
||||
image_data=request._input_image_data,
|
||||
audio_data=None,
|
||||
video_data=None,
|
||||
modalities=[],
|
||||
@@ -1540,6 +1530,64 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
result.reasoning_end_token_ids = list(token_ids)
|
||||
return result
|
||||
|
||||
def _apply_request_defaults(self, request: ChatCompletionRequest) -> None:
|
||||
"""Settle the thinking and effort that rendering reads; repeating it
|
||||
changes nothing."""
|
||||
if self.default_chat_template_kwargs:
|
||||
ctk = dict(request.chat_template_kwargs or {})
|
||||
for k, v in self.default_chat_template_kwargs.items():
|
||||
ctk.setdefault(k, v)
|
||||
request.chat_template_kwargs = ctk
|
||||
effort = ctk.get("reasoning_effort")
|
||||
if effort is not None and request.reasoning_effort is None:
|
||||
request.reasoning_effort = effort
|
||||
|
||||
normalize_hunyuan_reasoning_effort(
|
||||
request, self.reasoning_parser, self.template_manager.reasoning_config
|
||||
)
|
||||
|
||||
@property
|
||||
def native_tool_loading(self) -> bool:
|
||||
"""Whether Responses and Messages requests that load tools mid-conversation
|
||||
render from their own wire form (see `prepare_native_prompt`)."""
|
||||
return (
|
||||
self._encoding_adapter is not None
|
||||
and self._encoding_adapter.native_encoder is not None
|
||||
)
|
||||
|
||||
def prepare_native_prompt(
|
||||
self,
|
||||
request: ChatCompletionRequest,
|
||||
*,
|
||||
protocol: chat_encoding_adapters.NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
) -> chat_encoding_adapters.NativePrompt:
|
||||
"""Render `body`, the Responses or Messages form of `request`, into its
|
||||
`input_ids` and images, and set its `tools` to the tools the model can call
|
||||
after the history; rendering then keeps only grammar, stops and parsing."""
|
||||
self._apply_request_defaults(request)
|
||||
response_format = request.response_format
|
||||
prompt = self._encoding_adapter.native_encoder.encode_native(
|
||||
protocol=protocol,
|
||||
body=body,
|
||||
thinking=self._get_reasoning_from_request(request),
|
||||
effort=request.reasoning_effort,
|
||||
response_schema=(
|
||||
response_format.json_schema.schema_
|
||||
if response_format is not None and response_format.json_schema
|
||||
else None
|
||||
),
|
||||
continue_final_message=request.continue_final_message,
|
||||
)
|
||||
if prompt.image_data and not self.tokenizer_manager.model_config.is_multimodal:
|
||||
raise ValueError("image input is not supported for this model")
|
||||
request.input_ids = prompt.prompt_ids
|
||||
request.tools = prompt.tools or None
|
||||
request._input_image_data = prompt.image_data or None
|
||||
request._drops_unlisted_tool_calls = True
|
||||
request._continued_turn = prompt.continued_turn
|
||||
return prompt
|
||||
|
||||
def _apply_jinja_template(
|
||||
self,
|
||||
request: ChatCompletionRequest,
|
||||
|
||||
@@ -5,15 +5,20 @@ request semantics."""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any, Optional, Union
|
||||
|
||||
from sglang.srt.entrypoints.openai.chat_encoding_adapters import (
|
||||
ADAPTER_OUTPUT_PARSERS,
|
||||
NativePrompt,
|
||||
NativeProtocol,
|
||||
select_new_output_ids,
|
||||
token_output_sse_chunks,
|
||||
)
|
||||
from sglang.srt.entrypoints.openai.protocol import (
|
||||
Function,
|
||||
FunctionResponse,
|
||||
Tool,
|
||||
ToolCall,
|
||||
)
|
||||
from sglang.srt.environ import envs
|
||||
@@ -26,7 +31,9 @@ from sglang.srt.parser.deepseek_recipe_output import (
|
||||
whole_tool_calls,
|
||||
)
|
||||
from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
BUDGET_TIER,
|
||||
DeepSeekRecipe,
|
||||
loaded_tools,
|
||||
parse_effort_budget,
|
||||
)
|
||||
from sglang.srt.runtime_context import get_serving
|
||||
@@ -37,7 +44,6 @@ if TYPE_CHECKING:
|
||||
from sglang.srt.entrypoints.openai.protocol import (
|
||||
ChatCompletionRequest,
|
||||
ResponsesRequest,
|
||||
Tool,
|
||||
)
|
||||
|
||||
DEFAULT_EFFORT_ENVS = {
|
||||
@@ -45,6 +51,12 @@ DEFAULT_EFFORT_ENVS = {
|
||||
"dsv41": envs.SGLANG_DSV41_REASONING_EFFORT,
|
||||
}
|
||||
JSON_RESPONSE_FORMATS = ("json_object", "json_schema")
|
||||
# V4 renders deferred tools up front and never loads one mid-conversation.
|
||||
NATIVE_TOOL_LOADING_SPEC = "dsv41"
|
||||
NATIVE_EFFORT_FIELDS: dict[str, str] = {
|
||||
"responses": "reasoning",
|
||||
"messages": "output_config",
|
||||
}
|
||||
|
||||
|
||||
class DeepSeekRecipeChatEncodingAdapter:
|
||||
@@ -96,6 +108,13 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
def output(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
|
||||
return self if self._token_output else None
|
||||
|
||||
def validate_request(self, request: ChatCompletionRequest) -> Optional[str]:
|
||||
return None
|
||||
|
||||
@property
|
||||
def native_encoder(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
|
||||
return self if self._spec == NATIVE_TOOL_LOADING_SPEC else None
|
||||
|
||||
def encode_messages(
|
||||
self,
|
||||
*,
|
||||
@@ -149,6 +168,66 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
prompt_ids += self.deepseek_recipe.encode_text(task_prefix)
|
||||
return prompt_ids
|
||||
|
||||
def encode_native(
|
||||
self,
|
||||
*,
|
||||
protocol: NativeProtocol,
|
||||
body: dict[str, Any],
|
||||
thinking: bool,
|
||||
effort: Union[str, int, float, None],
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> NativePrompt:
|
||||
tier, budget = self._effort(effort)
|
||||
# recipe reads an effort as a request to think.
|
||||
if thinking and tier is not None:
|
||||
body = {**body, NATIVE_EFFORT_FIELDS[protocol]: {"effort": tier}}
|
||||
conversation = self.deepseek_recipe.with_host_settings(
|
||||
self.deepseek_recipe.convert_native(protocol, body, thinking=thinking),
|
||||
response_schema=response_schema,
|
||||
continue_final_message=continue_final_message,
|
||||
)
|
||||
last = conversation.messages[-1] if conversation.messages else None
|
||||
# recipe's Messages conversion drops `strict`, which SGLang's grammar enforces.
|
||||
strict = {tool.get("name") for tool in body["tools"] if tool.get("strict")}
|
||||
tools = [
|
||||
Tool(
|
||||
type="function",
|
||||
function=Function(
|
||||
name=tool.name,
|
||||
description=tool.description,
|
||||
parameters=tool.parameters,
|
||||
strict=True if tool.name in strict else tool.strict,
|
||||
),
|
||||
)
|
||||
for tool in loaded_tools(conversation)
|
||||
]
|
||||
return NativePrompt(
|
||||
prompt_ids=self.deepseek_recipe.encode_conversation(
|
||||
conversation, effort_budget=budget
|
||||
),
|
||||
tools=tools,
|
||||
image_data=[
|
||||
ImageData(url=url)
|
||||
for url in self.deepseek_recipe.image_urls(conversation)
|
||||
],
|
||||
namespace_names=(
|
||||
self.deepseek_recipe.namespace_tool_names(_responses_namespaces(body))
|
||||
if protocol == "responses"
|
||||
else {}
|
||||
),
|
||||
continued_turn=(
|
||||
{"role": "assistant", "content": last.content}
|
||||
| (
|
||||
{"reasoning_content": last.reasoning_content}
|
||||
if last.reasoning_content is not None
|
||||
else {}
|
||||
)
|
||||
if last is not None and last.prefix
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _encode(
|
||||
self,
|
||||
body: dict[str, Any],
|
||||
@@ -157,21 +236,29 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
task: Optional[str] = None,
|
||||
image_urls: Optional[list[str]] = None,
|
||||
) -> list[int]:
|
||||
if effort is None or effort == "none":
|
||||
effort = self._default_effort
|
||||
budget = None
|
||||
if effort is not None and not isinstance(effort, str):
|
||||
if self._spec != "dsv41":
|
||||
raise ValueError(
|
||||
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
|
||||
)
|
||||
budget = parse_effort_budget(effort)
|
||||
elif effort is not None:
|
||||
body["reasoning_effort"] = effort
|
||||
tier, budget = self._effort(effort)
|
||||
if budget is None and tier is not None:
|
||||
body["reasoning_effort"] = tier
|
||||
return self.deepseek_recipe.encode(
|
||||
body, task=task, effort_budget=budget, image_urls=image_urls
|
||||
)
|
||||
|
||||
def _effort(
|
||||
self, effort: Union[str, int, float, None]
|
||||
) -> tuple[Optional[str], Optional[int]]:
|
||||
"""The named tier recipe renders and, for a V4.1 integer budget, the budget
|
||||
that replaces the tier's line."""
|
||||
effort = _numeric_effort(effort)
|
||||
if effort is None or effort == "none":
|
||||
effort = self._default_effort
|
||||
if effort is None or isinstance(effort, str):
|
||||
return effort, None
|
||||
if self._spec != "dsv41":
|
||||
raise ValueError(
|
||||
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
|
||||
)
|
||||
return BUDGET_TIER, parse_effort_budget(effort)
|
||||
|
||||
def parse_chat_response(
|
||||
self,
|
||||
*,
|
||||
@@ -320,7 +407,7 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
self, request: ChatCompletionRequest
|
||||
) -> DeepSeekRecipeOutputParser:
|
||||
response_format = request.response_format
|
||||
continued = (
|
||||
continued = request._continued_turn or (
|
||||
_trailing_assistant_turn(
|
||||
[
|
||||
message.model_dump(exclude_unset=True, by_alias=True, mode="json")
|
||||
@@ -344,6 +431,11 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
separate_reasoning=self._separate_reasoning and request.separate_reasoning,
|
||||
stream_reasoning=request.stream_reasoning,
|
||||
stop=_stop_strings(request.stop),
|
||||
callable_names=(
|
||||
_native_callable_names(request.tools or [], request.tool_choice)
|
||||
if request._drops_unlisted_tool_calls
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _new_responses_parser(
|
||||
@@ -353,12 +445,25 @@ class DeepSeekRecipeChatEncodingAdapter:
|
||||
separate_reasoning: bool,
|
||||
parse_tool_calls: bool,
|
||||
) -> DeepSeekRecipeOutputParser:
|
||||
native = request._tool_state.chat_tools is not None
|
||||
return DeepSeekRecipeOutputParser(
|
||||
self.deepseek_recipe,
|
||||
thinking_stage=self.deepseek_recipe.parse_start(thinking=require_reasoning),
|
||||
parse_tool_calls=parse_tool_calls,
|
||||
# Under `none` the tool parser runs only to drop calls; a JSON answer
|
||||
# must be read as JSON, or a call marker spelled in a string ends it.
|
||||
parse_json_output=native
|
||||
and request.tool_choice == "none"
|
||||
and request.has_json_schema_constraint(),
|
||||
separate_reasoning=separate_reasoning,
|
||||
stop=_stop_strings(request.stop),
|
||||
callable_names=(
|
||||
_native_callable_names(
|
||||
request._tool_state.chat_tools, request.tool_choice
|
||||
)
|
||||
if native
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
def _tool_call_id(self, index: int, name: str, history_tool_calls_cnt: int) -> str:
|
||||
@@ -390,6 +495,37 @@ class DeepSeekRecipeResponsesStream:
|
||||
return delta.reasoning, delta.content, _tool_call_items(delta.tool_calls)
|
||||
|
||||
|
||||
def _responses_namespaces(body: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""The namespace tools a Responses body declares and its history discovers, in
|
||||
the order recipe's conversion names their members."""
|
||||
discovered = [
|
||||
tool
|
||||
for item in body["input"]
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
for tool in item["tools"]
|
||||
]
|
||||
return [
|
||||
tool for tool in [*body["tools"], *discovered] if tool["type"] == "namespace"
|
||||
]
|
||||
|
||||
|
||||
def _null_content_as_empty(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""As `_process_messages` renders them: a null content is ""."""
|
||||
return [
|
||||
message if message.get("content") is not None else {**message, "content": ""}
|
||||
for message in messages
|
||||
]
|
||||
|
||||
|
||||
def _numeric_effort(
|
||||
effort: Union[str, int, float, None],
|
||||
) -> Union[str, int, float, None]:
|
||||
"""A numeric string effort ("30", "0.3") as the number it spells."""
|
||||
if isinstance(effort, str) and re.fullmatch(r"\d+(\.\d+)?", effort.strip()):
|
||||
return float(effort) if "." in effort else int(effort)
|
||||
return effort
|
||||
|
||||
|
||||
def _trailing_assistant_turn(
|
||||
messages: list[dict[str, Any]],
|
||||
) -> Optional[dict[str, Any]]:
|
||||
@@ -405,6 +541,18 @@ def _stop_strings(stop: Union[str, list[str], None]) -> list[str]:
|
||||
return [stop] if isinstance(stop, str) else stop or []
|
||||
|
||||
|
||||
def _native_callable_names(
|
||||
tools: list[Tool], tool_choice: Any
|
||||
) -> Optional[frozenset[str]]:
|
||||
"""The names of `tools`, all that a natively rendered prompt shows the model;
|
||||
None when calls to other tools are returned. `none` allows no call at all."""
|
||||
if tool_choice == "none":
|
||||
return frozenset()
|
||||
if envs.SGLANG_FORWARD_UNKNOWN_TOOLS.get():
|
||||
return None
|
||||
return frozenset(tool.function.name for tool in tools)
|
||||
|
||||
|
||||
def _tool_call_items(
|
||||
calls: Sequence[Union[RecipeToolCall, RecipeToolCallDelta]],
|
||||
) -> list[ToolCallItem]:
|
||||
|
||||
@@ -36,7 +36,10 @@ if TYPE_CHECKING:
|
||||
|
||||
class InklingChatEncodingAdapter:
|
||||
collects_openai_media = True
|
||||
strict_tool_call_arguments = True
|
||||
history_tool_call_arguments = "strict"
|
||||
renders_response_format = False
|
||||
uncounted_prompt_tokens = 0
|
||||
native_encoder = None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
||||
@@ -13,6 +13,7 @@ from http import HTTPStatus
|
||||
from typing import TYPE_CHECKING, Any, AsyncGenerator, AsyncIterator, Optional, Union
|
||||
|
||||
import jinja2
|
||||
import msgspec
|
||||
import openai.types.responses as openai_responses_types
|
||||
import orjson
|
||||
from fastapi import Request
|
||||
@@ -43,6 +44,7 @@ from openai.types.responses.response_reasoning_summary_part_done_event import (
|
||||
Part as ResponseReasoningSummaryDonePart,
|
||||
)
|
||||
from openai_harmony import Message as OpenAIMessage
|
||||
from pydantic import BaseModel, TypeAdapter
|
||||
|
||||
from sglang.srt.entrypoints.context import (
|
||||
ConversationContext,
|
||||
@@ -70,7 +72,12 @@ from sglang.srt.entrypoints.openai.protocol import (
|
||||
ResponseOutputMessage,
|
||||
ResponsesRequest,
|
||||
ResponsesResponse,
|
||||
ResponseTool,
|
||||
ResponseToolSearchCall,
|
||||
ResponseToolSearchCallEvent,
|
||||
Tool,
|
||||
ToolChoice,
|
||||
ToolChoiceFuncName,
|
||||
UsageInfo,
|
||||
)
|
||||
from sglang.srt.entrypoints.openai.responses_adapters import (
|
||||
@@ -114,6 +121,100 @@ class _MediaInputValidationError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class ResponseToolChoiceError(ValueError):
|
||||
"""A response tool_choice that names a tool the model cannot call."""
|
||||
|
||||
|
||||
TOOL_SEARCH = "tool_search"
|
||||
TOOL_SEARCH_ITEMS = ("tool_search_call", "tool_search_output")
|
||||
APPLY_PATCH = "apply_patch"
|
||||
NAMESPACE_MEMBERS_ERROR = (
|
||||
"namespace tools must list function tools, each with a type and a name"
|
||||
)
|
||||
|
||||
|
||||
def _holds_function_tools(members: Any) -> bool:
|
||||
return isinstance(members, list) and all(
|
||||
isinstance(member, dict)
|
||||
and member.get("type") == "function"
|
||||
and isinstance(member.get("name"), str)
|
||||
for member in members
|
||||
)
|
||||
|
||||
|
||||
def _defers_loading(tool: ResponseTool) -> bool:
|
||||
if tool.type == "namespace":
|
||||
return bool(tool.defer_loading) or any(
|
||||
isinstance(member, dict) and member.get("defer_loading")
|
||||
for member in tool.tools or []
|
||||
)
|
||||
return tool.type in ("function", "custom") and bool(tool.defer_loading)
|
||||
|
||||
|
||||
def _defer_tool_loading(
|
||||
tools: list[dict[str, Any]], items: list
|
||||
) -> tuple[list[dict[str, Any]], list]:
|
||||
"""`tools` without those that defer loading, and `items` whose
|
||||
`tool_search_output` entries naming a deferred tool take the fields they
|
||||
leave out from its declaration, so a search result loads it where it returns.
|
||||
recipe's Responses conversion renders a deferred tool up front and rejects a
|
||||
search result that repeats a declared name."""
|
||||
deferred: dict[tuple[Optional[str], str], dict[str, Any]] = {}
|
||||
shown = []
|
||||
for tool in tools:
|
||||
if tool["type"] == "namespace":
|
||||
members = []
|
||||
for member in tool["tools"]:
|
||||
if tool.get("defer_loading") or member.get("defer_loading"):
|
||||
deferred[(tool["name"], member["name"])] = member
|
||||
else:
|
||||
members.append(member)
|
||||
if members:
|
||||
shown.append({**tool, "tools": members})
|
||||
elif tool["type"] in ("function", "custom") and tool.get("defer_loading"):
|
||||
deferred[(None, tool["name"])] = tool
|
||||
else:
|
||||
shown.append(tool)
|
||||
|
||||
def loaded(tool: dict[str, Any], namespace: Optional[str] = None) -> dict:
|
||||
if tool["type"] == "namespace":
|
||||
members = [loaded(member, tool["name"]) for member in tool["tools"]]
|
||||
return {**tool, "tools": members}
|
||||
declared = deferred.get((namespace, tool.get("name")))
|
||||
if declared is None or declared["type"] != tool["type"]:
|
||||
return tool
|
||||
return declared | {k: v for k, v in tool.items() if v is not None}
|
||||
|
||||
if deferred:
|
||||
items = [
|
||||
(
|
||||
{**item, "tools": [loaded(tool) for tool in item["tools"]]}
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
else item
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
return shown, items
|
||||
|
||||
|
||||
def _tool_search_call(
|
||||
item_id: str, call_id: str, arguments: str
|
||||
) -> ResponseToolSearchCall:
|
||||
"""A search call reports object arguments; one whose arguments are not a JSON
|
||||
object is incomplete."""
|
||||
try:
|
||||
parsed = json.loads(arguments)
|
||||
except ValueError:
|
||||
parsed = None
|
||||
complete = isinstance(parsed, dict)
|
||||
return ResponseToolSearchCall(
|
||||
id=item_id,
|
||||
call_id=call_id,
|
||||
arguments=parsed if complete else {},
|
||||
status="completed" if complete else "incomplete",
|
||||
)
|
||||
|
||||
|
||||
def _build_output_text_logprobs(meta_info: dict) -> list[Logprob]:
|
||||
"""Reshape decoded ``meta_info`` logprobs into the Responses logprob type,
|
||||
covering every generated token."""
|
||||
@@ -319,9 +420,21 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
# FIXME: If the engine is dead, raise an error
|
||||
# This is required for the streaming case
|
||||
|
||||
# Such a request's callable tools are known once its history renders,
|
||||
# so `_make_request` checks its tool_choice instead.
|
||||
native = self._renders_natively(request)
|
||||
if native:
|
||||
tools_error = self._native_tools_error(request)
|
||||
if tools_error is not None:
|
||||
return self.create_error_response(tools_error, param="tools")
|
||||
|
||||
# ``tool_choice="required"`` needs a tool the parser can actually force.
|
||||
if request.tool_choice == "required" and not any(
|
||||
tool.type in ("function", "custom") for tool in (request.tools or [])
|
||||
if (
|
||||
not native
|
||||
and request.tool_choice == "required"
|
||||
and not any(
|
||||
tool.type in ("function", "custom") for tool in (request.tools or [])
|
||||
)
|
||||
):
|
||||
return self.create_error_response(
|
||||
'tool_choice="required" requires at least one tool with '
|
||||
@@ -330,9 +443,13 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
|
||||
tool_choice = request.effective_tool_choice()
|
||||
if isinstance(tool_choice, dict) and not any(
|
||||
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
|
||||
for tool in request.tools or []
|
||||
if (
|
||||
not native
|
||||
and isinstance(tool_choice, dict)
|
||||
and not any(
|
||||
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
|
||||
for tool in request.tools or []
|
||||
)
|
||||
):
|
||||
return self.create_error_response(
|
||||
f"Tool {tool_choice['name']!r} is not declared in tools",
|
||||
@@ -415,6 +532,8 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
|
||||
except _MediaInputValidationError as e:
|
||||
return self.create_error_response(str(e))
|
||||
except ResponseToolChoiceError as e:
|
||||
return self.create_error_response(str(e), param="tool_choice")
|
||||
except (ValueError, TypeError, RuntimeError, jinja2.TemplateError) as e:
|
||||
logger.exception("Error in preprocessing prompt inputs")
|
||||
return self.create_error_response(f"{e} {e.__cause__}")
|
||||
@@ -684,7 +803,10 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
prev_response: Optional[ResponsesResponse],
|
||||
tokenizer: Any,
|
||||
):
|
||||
messages = self._construct_input_messages(request, prev_response)
|
||||
native = self._renders_natively(request)
|
||||
messages = (
|
||||
[] if native else self._construct_input_messages(request, prev_response)
|
||||
)
|
||||
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
chat_request = ChatCompletionRequest(
|
||||
@@ -707,6 +829,8 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
chat_template_kwargs=request.chat_template_kwargs,
|
||||
)
|
||||
chat_request._template_tools = self._response_tools_to_template_tools(request)
|
||||
if native:
|
||||
self._prepare_native_responses_prompt(request, chat_request)
|
||||
|
||||
media_error = self._validate_media_content(chat_request)
|
||||
if media_error:
|
||||
@@ -906,10 +1030,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
else:
|
||||
self.msg_store[response.id] = self._response_input_history(
|
||||
request
|
||||
) + [
|
||||
item.model_dump(exclude_none=True)
|
||||
for item in response.output
|
||||
]
|
||||
) + self._replayed_output_items(request, response.output)
|
||||
|
||||
return response
|
||||
|
||||
@@ -952,12 +1073,18 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
name: str,
|
||||
arguments: str,
|
||||
custom_names: set[str],
|
||||
namespace_names: Optional[dict[str, tuple[str, str]]] = None,
|
||||
) -> Union[ResponseFunctionToolCall, ResponseCustomToolCall]:
|
||||
tool_names: dict[str, dict[str, str]],
|
||||
*,
|
||||
client_tool_search: bool = False,
|
||||
) -> Union[
|
||||
ResponseFunctionToolCall, ResponseCustomToolCall, ResponseToolSearchCall
|
||||
]:
|
||||
"""A call against a ``custom`` tool reports its freeform payload rather
|
||||
than the JSON arguments of the shim function tool. A call against a
|
||||
namespace member reports ``namespace`` and the bare function ``name``."""
|
||||
call_id = f"call_{random_uuid()[:24]}"
|
||||
if client_tool_search and name == TOOL_SEARCH:
|
||||
return _tool_search_call(f"tsc_{random_uuid()[:8]}", call_id, arguments)
|
||||
if name in custom_names:
|
||||
return ResponseCustomToolCall(
|
||||
type="custom_tool_call",
|
||||
@@ -966,7 +1093,13 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
name=name,
|
||||
input=decode_custom_tool_input(arguments),
|
||||
)
|
||||
namespace, name = (namespace_names or {}).get(name, (None, name))
|
||||
entry = (tool_names or {}).get(name)
|
||||
if isinstance(entry, tuple):
|
||||
namespace, name = entry
|
||||
elif isinstance(entry, dict):
|
||||
namespace, name = entry.get("namespace"), entry.get("name", name)
|
||||
else:
|
||||
namespace = None
|
||||
return ResponseFunctionToolCall(
|
||||
arguments=arguments,
|
||||
call_id=call_id,
|
||||
@@ -1063,12 +1196,17 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
finish_reason: Optional[dict[str, Any]] = None,
|
||||
):
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
custom_names = custom_tool_names(request.tools)
|
||||
custom_names = self._custom_tool_names(request)
|
||||
namespace_names = (
|
||||
namespace_function_names(request.tools)
|
||||
if self.tool_call_parser == "muse"
|
||||
else {}
|
||||
else {
|
||||
call_name: (entry.get("namespace"), entry.get("name", call_name))
|
||||
for call_name, entry in request._tool_state.names.items()
|
||||
if entry.get("namespace")
|
||||
}
|
||||
)
|
||||
client_tool_search = self._client_tool_search(request)
|
||||
encoding_output = self._encoding_output()
|
||||
if encoding_output is not None:
|
||||
required_json = self._uses_required_tool_json(request, chat_tools)
|
||||
@@ -1088,12 +1226,17 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
content, custom_names, namespace_names
|
||||
)
|
||||
else:
|
||||
tool_call_items = [
|
||||
self._make_tool_call_item(
|
||||
call.name, call.parameters, custom_names, namespace_names
|
||||
tool_call_items = []
|
||||
for call in calls:
|
||||
item = self._make_tool_call_item(
|
||||
call.name,
|
||||
call.parameters,
|
||||
custom_names,
|
||||
namespace_names,
|
||||
client_tool_search=client_tool_search,
|
||||
)
|
||||
for call in calls
|
||||
]
|
||||
self._record_called_name(request, item.call_id, call.name)
|
||||
tool_call_items.append(item)
|
||||
else:
|
||||
reasoning_content, content, tool_call_items = self._parse_text_output(
|
||||
request,
|
||||
@@ -1194,14 +1337,15 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
try:
|
||||
content, call_info_list = parser.parse_non_stream(content)
|
||||
for call_info in call_info_list:
|
||||
tool_call_items.append(
|
||||
self._make_tool_call_item(
|
||||
call_info.name,
|
||||
call_info.parameters or "",
|
||||
custom_names,
|
||||
namespace_names,
|
||||
)
|
||||
item = self._make_tool_call_item(
|
||||
call_info.name,
|
||||
call_info.parameters or "",
|
||||
custom_names,
|
||||
namespace_names,
|
||||
client_tool_search=self._client_tool_search(request),
|
||||
)
|
||||
self._record_called_name(request, item.call_id, call_info.name)
|
||||
tool_call_items.append(item)
|
||||
parsed_via_native = bool(call_info_list)
|
||||
except Exception as e:
|
||||
logger.error("Tool call parsing error: %s", e)
|
||||
@@ -1268,6 +1412,10 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
def _tool_call_parsing_enabled(
|
||||
self, request: ResponsesRequest, chat_tools: list[Tool]
|
||||
) -> bool:
|
||||
if request._tool_state.chat_tools is not None and request.tool_choice == "none":
|
||||
# The history showed the model tools, so it can still write a call;
|
||||
# parsing drops it, where skipping the parser would return its markup.
|
||||
return bool(self.tool_call_parser)
|
||||
return bool(
|
||||
chat_tools and self.tool_call_parser and request.tool_choice != "none"
|
||||
)
|
||||
@@ -1326,7 +1474,213 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
return tool_choice
|
||||
return {"type": "function", "function": {"name": tool_choice["name"]}}
|
||||
|
||||
def _renders_natively(self, request: ResponsesRequest) -> bool:
|
||||
"""Whether the served encoding renders the request in its own wire form,
|
||||
which tool search, namespace tools and deferred tools take."""
|
||||
return self.native_tool_loading and (
|
||||
any(
|
||||
tool.type == "namespace" or _defers_loading(tool)
|
||||
for tool in request.tools
|
||||
)
|
||||
or self._uses_tool_search(request)
|
||||
)
|
||||
|
||||
def _uses_tool_search(self, request: ResponsesRequest) -> bool:
|
||||
"""Whether the request searches for tools, or its history has."""
|
||||
if any(tool.type == TOOL_SEARCH for tool in request.tools):
|
||||
return True
|
||||
return any(
|
||||
isinstance(item, dict) and item.get("type") in TOOL_SEARCH_ITEMS
|
||||
for item in self._history_items(request)
|
||||
)
|
||||
|
||||
def _history_items(self, request: ResponsesRequest) -> list:
|
||||
"""The input history, before `previous_response_id` is known to exist."""
|
||||
stored = self.msg_store.get(request.previous_response_id) or []
|
||||
return list(stored) + (request.input if isinstance(request.input, list) else [])
|
||||
|
||||
def _native_tools_error(self, request: ResponsesRequest) -> Optional[str]:
|
||||
deferred_types = {
|
||||
tool.name: tool.type
|
||||
for tool in request.tools
|
||||
if tool.type in ("function", "custom") and tool.defer_loading
|
||||
}
|
||||
for tool in request.tools:
|
||||
if (
|
||||
tool.type == TOOL_SEARCH
|
||||
and (tool.model_extra or {}).get("execution") != "client"
|
||||
):
|
||||
return (
|
||||
'tool_search requires execution "client": this server runs no '
|
||||
"searches"
|
||||
)
|
||||
if tool.type == "namespace" and not _holds_function_tools(tool.tools):
|
||||
return NAMESPACE_MEMBERS_ERROR
|
||||
for item in self._history_items(request):
|
||||
if not isinstance(item, dict) or item.get("type") != "tool_search_output":
|
||||
continue
|
||||
tools = item.get("tools")
|
||||
if not isinstance(tools, list) or not all(
|
||||
isinstance(tool, dict)
|
||||
and tool.get("type") in ("function", "namespace", "custom")
|
||||
for tool in tools
|
||||
):
|
||||
return (
|
||||
"tool_search_output tools must be a list of function, namespace "
|
||||
"or custom tools"
|
||||
)
|
||||
for tool in tools:
|
||||
if tool["type"] == "namespace" and not _holds_function_tools(
|
||||
tool.get("tools")
|
||||
):
|
||||
return NAMESPACE_MEMBERS_ERROR
|
||||
# The served encoding renders no other custom tool.
|
||||
if tool["type"] == "custom" and tool.get("name") != APPLY_PATCH:
|
||||
return (
|
||||
f"tool_search_output can load only the {APPLY_PATCH!r} "
|
||||
"custom tool"
|
||||
)
|
||||
# A deferred declaration is loaded as the tool it declares.
|
||||
declared = deferred_types.get(tool.get("name"), tool["type"])
|
||||
if tool["type"] != "namespace" and declared != tool["type"]:
|
||||
return (
|
||||
f"tool_search_output returns {tool.get('name')!r} as a "
|
||||
f"{tool['type']} tool, but tools declares it as a deferred "
|
||||
f"{declared} tool"
|
||||
)
|
||||
return None
|
||||
|
||||
def _prepare_native_responses_prompt(
|
||||
self, request: ResponsesRequest, chat_request: ChatCompletionRequest
|
||||
) -> None:
|
||||
"""Render the request's own wire form into `chat_request`, whose tools and
|
||||
choice become the ones the history has loaded."""
|
||||
choice = request.effective_tool_choice()
|
||||
tools, items = _defer_tool_loading(
|
||||
[tool.model_dump(exclude_unset=True) for tool in request.tools],
|
||||
[
|
||||
item
|
||||
for item in map(
|
||||
self._native_input_item, self._response_input_history(request)
|
||||
)
|
||||
if item is not None
|
||||
],
|
||||
)
|
||||
body = {"input": items, "tools": tools}
|
||||
if request.instructions:
|
||||
body["instructions"] = request.instructions
|
||||
if choice == "none":
|
||||
body["tool_choice"] = "none"
|
||||
prompt = self.prepare_native_prompt(
|
||||
chat_request, protocol="responses", body=body
|
||||
)
|
||||
tools = prompt.tools
|
||||
request._tool_state = msgspec.structs.replace(
|
||||
request._tool_state,
|
||||
chat_tools=tools,
|
||||
names={
|
||||
call_name: {"namespace": namespace, "name": name}
|
||||
for call_name, (namespace, name) in prompt.namespace_names.items()
|
||||
},
|
||||
discovered_custom_names=frozenset(
|
||||
tool["name"]
|
||||
for item in body["input"]
|
||||
if isinstance(item, dict) and item.get("type") == "tool_search_output"
|
||||
for tool in item["tools"]
|
||||
if tool["type"] == "custom"
|
||||
),
|
||||
)
|
||||
if isinstance(choice, dict):
|
||||
name = choice["name"]
|
||||
if "namespace" in choice:
|
||||
# `tools` keeps conversion order, declared before discovered, so
|
||||
# the last match is the member's latest definition.
|
||||
member = (choice["namespace"], name)
|
||||
name = next(
|
||||
(
|
||||
tool.function.name
|
||||
for tool in reversed(tools)
|
||||
if prompt.namespace_names.get(tool.function.name) == member
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not any(tool.function.name == name for tool in tools) or (
|
||||
choice["type"] == "custom"
|
||||
and name not in self._custom_tool_names(request)
|
||||
):
|
||||
raise ResponseToolChoiceError(
|
||||
f"Tool {choice['name']!r} is not declared in tools without "
|
||||
"defer_loading or loaded by a tool_search_output"
|
||||
)
|
||||
chat_request.tool_choice = ToolChoice(
|
||||
function=ToolChoiceFuncName(name=name)
|
||||
)
|
||||
elif choice == "required" and not tools:
|
||||
raise ResponseToolChoiceError(
|
||||
'tool_choice="required" needs a tool the model can call'
|
||||
)
|
||||
else:
|
||||
chat_request.tool_choice = choice if tools else "none"
|
||||
|
||||
@classmethod
|
||||
def _native_input_item(cls, item: Any) -> Any:
|
||||
if isinstance(item, BaseModel):
|
||||
item = item.model_dump(exclude_none=True)
|
||||
if isinstance(item, dict) and item.get("type") == "reasoning":
|
||||
# recipe reads only `content`; keep the text SGLang replays from a
|
||||
# summary or a store=false client's state.
|
||||
message = cls._normalize_response_message_for_chat(item)
|
||||
if message is None:
|
||||
return None
|
||||
return {
|
||||
"type": "reasoning",
|
||||
"content": [
|
||||
{"type": "reasoning_text", "text": message["reasoning_content"]}
|
||||
],
|
||||
}
|
||||
return item
|
||||
|
||||
@staticmethod
|
||||
def _record_called_name(request: ResponsesRequest, call_id: str, name: str) -> None:
|
||||
"""Keep the name the model called a namespace member by: the item reports
|
||||
its namespace and own name, which several call names can share."""
|
||||
if request._tool_state.chat_tools is not None and (
|
||||
name in request._tool_state.names
|
||||
):
|
||||
request._called_tool_names[call_id] = name
|
||||
|
||||
@staticmethod
|
||||
def _replayed_output_items(request: ResponsesRequest, output: list) -> list[dict]:
|
||||
"""`output` as the history a later request replays: a namespace member's
|
||||
call under the name the model called it by."""
|
||||
items = []
|
||||
for item in output:
|
||||
item = item.model_dump(exclude_none=True)
|
||||
called_name = request._called_tool_names.get(item.get("call_id"))
|
||||
if called_name is not None:
|
||||
item = {**item, "name": called_name}
|
||||
del item["namespace"]
|
||||
items.append(item)
|
||||
return items
|
||||
|
||||
@staticmethod
|
||||
def _custom_tool_names(request: ResponsesRequest) -> set[str]:
|
||||
declared = custom_tool_names(request.tools)
|
||||
if request._tool_state.chat_tools is not None:
|
||||
# A deferred declaration counts once a search result has loaded it.
|
||||
declared -= {tool.name for tool in request.tools if tool.defer_loading}
|
||||
return declared | request._tool_state.discovered_custom_names
|
||||
|
||||
@staticmethod
|
||||
def _client_tool_search(request: ResponsesRequest) -> bool:
|
||||
"""Whether a `tool_search` call is the client-executed search tool."""
|
||||
return request._tool_state.chat_tools is not None and any(
|
||||
tool.type == TOOL_SEARCH for tool in request.tools
|
||||
)
|
||||
|
||||
def _response_tools_to_chat_tools(self, request: ResponsesRequest) -> list[Tool]:
|
||||
if request._tool_state.chat_tools is not None:
|
||||
return list(request._tool_state.chat_tools)
|
||||
# ``function`` and ``custom`` tools flow to chat; built-ins go through
|
||||
# harmony. A custom tool is shimmed into a single-string function tool.
|
||||
chat_tools = []
|
||||
@@ -2133,6 +2487,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
tool_follows = opens_item and item_type in (
|
||||
"function_call",
|
||||
"custom_tool_call",
|
||||
"tool_search_call",
|
||||
)
|
||||
terminal = event_type in (
|
||||
"response.completed",
|
||||
@@ -2215,14 +2570,19 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
|
||||
chat_tools = self._response_tools_to_chat_tools(request)
|
||||
custom_names = custom_tool_names(request.tools)
|
||||
custom_names = self._custom_tool_names(request)
|
||||
namespace_names = (
|
||||
namespace_function_names(request.tools)
|
||||
if self.tool_call_parser == "muse"
|
||||
else {}
|
||||
else {
|
||||
call_name: (entry.get("namespace"), entry.get("name", call_name))
|
||||
for call_name, entry in request._tool_state.names.items()
|
||||
if entry.get("namespace")
|
||||
}
|
||||
)
|
||||
tool_choice = request.effective_tool_choice()
|
||||
required_json = self._uses_required_tool_json(request, chat_tools)
|
||||
client_tool_search = self._client_tool_search(request)
|
||||
encoding_output = self._encoding_output()
|
||||
token_stream = (
|
||||
encoding_output.new_responses_stream(
|
||||
@@ -2453,7 +2813,13 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
return []
|
||||
arguments = state["arguments"]
|
||||
events: list = []
|
||||
if state["custom"]:
|
||||
done_event = openai_responses_types.ResponseOutputItemDoneEvent
|
||||
if state["search"]:
|
||||
completed_item = _tool_search_call(
|
||||
state["item_id"], state["call_id"], arguments
|
||||
)
|
||||
done_event = ResponseToolSearchCallEvent
|
||||
elif state["custom"]:
|
||||
payload = decode_custom_tool_input(arguments)
|
||||
# Deltas cannot be retracted, so a payload that no longer
|
||||
# extends what was already streamed defers to the streamed text.
|
||||
@@ -2520,7 +2886,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
)
|
||||
events.append(
|
||||
_send_event(
|
||||
openai_responses_types.ResponseOutputItemDoneEvent(
|
||||
done_event(
|
||||
type="response.output_item.done",
|
||||
sequence_number=-1,
|
||||
output_index=state["output_index"],
|
||||
@@ -2717,27 +3083,43 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
current_output_index += 1
|
||||
name = call.name or ""
|
||||
is_custom = name in custom_names
|
||||
is_search = client_tool_search and name == TOOL_SEARCH
|
||||
namespace, name = namespace_names.get(name, (None, name))
|
||||
prefix = (
|
||||
"ctc" if is_custom else "tsc" if is_search else "fc"
|
||||
)
|
||||
state = {
|
||||
"item_id": (
|
||||
f"ctc_{random_uuid()[:8]}"
|
||||
if is_custom
|
||||
else f"fc_{random_uuid()[:8]}"
|
||||
),
|
||||
"item_id": f"{prefix}_{random_uuid()[:8]}",
|
||||
"call_id": f"call_{random_uuid()[:24]}",
|
||||
"output_index": current_output_index,
|
||||
"name": name,
|
||||
"namespace": namespace,
|
||||
"arguments": "",
|
||||
"custom": is_custom,
|
||||
"search": is_search,
|
||||
"payload": "",
|
||||
"added": False,
|
||||
"done": False,
|
||||
}
|
||||
tool_call_states[tool_index] = state
|
||||
self._record_called_name(
|
||||
request, state["call_id"], call.name or ""
|
||||
)
|
||||
if not state["added"]:
|
||||
state["added"] = True
|
||||
if state["custom"]:
|
||||
added_event = (
|
||||
openai_responses_types.ResponseOutputItemAddedEvent
|
||||
)
|
||||
if state["search"]:
|
||||
# Search arguments are reported whole when it completes.
|
||||
added_item = ResponseToolSearchCall(
|
||||
id=state["item_id"],
|
||||
call_id=state["call_id"],
|
||||
arguments={},
|
||||
status="in_progress",
|
||||
)
|
||||
added_event = ResponseToolSearchCallEvent
|
||||
elif state["custom"]:
|
||||
added_item = ResponseCustomToolCall(
|
||||
type="custom_tool_call",
|
||||
id=state["item_id"],
|
||||
@@ -2756,7 +3138,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
**_namespace_field(state),
|
||||
)
|
||||
yield _send_event(
|
||||
openai_responses_types.ResponseOutputItemAddedEvent(
|
||||
added_event(
|
||||
type="response.output_item.added",
|
||||
sequence_number=-1,
|
||||
output_index=state["output_index"],
|
||||
@@ -2786,7 +3168,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
delta=delta,
|
||||
)
|
||||
)
|
||||
else:
|
||||
elif not state["search"]:
|
||||
yield _send_event(
|
||||
openai_responses_types.ResponseFunctionCallArgumentsDeltaEvent(
|
||||
type="response.function_call_arguments.delta",
|
||||
@@ -2936,10 +3318,7 @@ class OpenAIServingResponses(OpenAIServingChat):
|
||||
self.response_store[final_response.id] = final_response
|
||||
self.msg_store[final_response.id] = self._response_input_history(
|
||||
request
|
||||
) + [
|
||||
item.model_dump(exclude_none=True)
|
||||
for item in final_response.output
|
||||
]
|
||||
) + self._replayed_output_items(request, final_response.output)
|
||||
|
||||
response_dict = _sanitize_response_dict(final_response.model_dump())
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ parses them, and tool-call arguments arrive as the API streams them."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from typing import Any, Optional
|
||||
|
||||
@@ -17,6 +18,11 @@ from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
load_deepseek_recipe,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# The most tokens one UTF-8 character can span, a byte each.
|
||||
_MAX_UTF8_BYTES = 4
|
||||
|
||||
|
||||
class RecipeToolCallDelta(msgspec.Struct, frozen=True):
|
||||
"""A piece of one call's JSON arguments; the first piece also names the call."""
|
||||
@@ -74,6 +80,7 @@ class DeepSeekRecipeOutputParser:
|
||||
separate_reasoning: bool = True,
|
||||
stream_reasoning: bool = True,
|
||||
stop: Sequence[str] = (),
|
||||
callable_names: Optional[frozenset[str]] = None,
|
||||
):
|
||||
recipe = load_deepseek_recipe()
|
||||
options = recipe.ParsingOptions(
|
||||
@@ -101,6 +108,10 @@ class DeepSeekRecipeOutputParser:
|
||||
# out, so a matched stop is cut before recipe has emitted any of it.
|
||||
self._stop_hold_chars = max(map(len, stop), default=1) - 1
|
||||
self._pending_ids: list[int] = []
|
||||
self._callable_names = callable_names
|
||||
# recipe's index of each call so far, mapped to the index it is reported
|
||||
# at; None for a dropped call.
|
||||
self._call_indices: dict[int, Optional[int]] = {}
|
||||
self.num_consumed_tokens = 0
|
||||
|
||||
def feed(self, token_ids: Sequence[int]) -> RecipeOutputDelta:
|
||||
@@ -192,6 +203,27 @@ class DeepSeekRecipeOutputParser:
|
||||
+ self._hf_tokenizer.encode(text[:cut], add_special_tokens=False).ids
|
||||
)
|
||||
|
||||
def _reported_call(
|
||||
self, call: RecipeToolCallDelta
|
||||
) -> Optional[RecipeToolCallDelta]:
|
||||
"""`call` at the index it is reported at; None when it belongs to a call
|
||||
outside `callable_names`."""
|
||||
if self._callable_names is None:
|
||||
return call
|
||||
if call.name is not None:
|
||||
index = None
|
||||
if call.name in self._callable_names:
|
||||
index = sum(kept is not None for kept in self._call_indices.values())
|
||||
else:
|
||||
logger.warning(
|
||||
"Model attempted to call undefined function: %s", call.name
|
||||
)
|
||||
self._call_indices[call.index] = index
|
||||
index = self._call_indices[call.index]
|
||||
if index is None:
|
||||
return None
|
||||
return msgspec.structs.replace(call, index=index)
|
||||
|
||||
def _deltas(self, chunks: list[Any]) -> RecipeOutputDelta:
|
||||
delta = RecipeOutputDelta()
|
||||
for chunk in chunks:
|
||||
@@ -202,12 +234,19 @@ class DeepSeekRecipeOutputParser:
|
||||
reasoning = piece.get("reasoning_content") or ""
|
||||
content = piece.get("content") or ""
|
||||
tool_calls = tuple(
|
||||
RecipeToolCallDelta(
|
||||
index=call["index"],
|
||||
arguments=call["function"]["arguments"],
|
||||
name=call["function"].get("name"),
|
||||
call
|
||||
for call in map(
|
||||
self._reported_call,
|
||||
(
|
||||
RecipeToolCallDelta(
|
||||
index=call["index"],
|
||||
arguments=call["function"]["arguments"],
|
||||
name=call["function"].get("name"),
|
||||
)
|
||||
for call in piece.get("tool_calls") or ()
|
||||
),
|
||||
)
|
||||
for call in piece.get("tool_calls") or ()
|
||||
if call is not None
|
||||
)
|
||||
if self._reasoning_as_content:
|
||||
content, reasoning = reasoning + content, ""
|
||||
|
||||
@@ -6,6 +6,7 @@ from __future__ import annotations
|
||||
import copy
|
||||
import functools
|
||||
import json
|
||||
import logging
|
||||
from types import ModuleType
|
||||
from typing import Any, Optional, Union
|
||||
|
||||
@@ -15,6 +16,8 @@ DEEPSEEK_RECIPE_INSTALL_HINT = (
|
||||
"`pip install deepseek-recipe`."
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MODEL_NAMES = {"dsv4": "DeepSeek-V4-Pro", "dsv41": "DeepSeek-V4.1-Flash"}
|
||||
# The checkpoint tokenizer's image token; recipe's bundled tokenizer spells it <|image|>.
|
||||
IMAGE_TOKEN = "<|deepseek_image|>"
|
||||
@@ -95,6 +98,23 @@ def pre_normalize(body: dict[str, Any]) -> dict[str, Any]:
|
||||
return request
|
||||
|
||||
|
||||
def loaded_tools(conversation: Any) -> list[Any]:
|
||||
"""The tools the model can call after `conversation`'s history: those it does
|
||||
not defer and the deferred ones a system message has loaded."""
|
||||
recipe = load_deepseek_recipe()
|
||||
loaded = {
|
||||
tool.name
|
||||
for message in conversation.messages
|
||||
if isinstance(message, recipe.SystemMessage)
|
||||
for tool in message.tools or []
|
||||
}
|
||||
return [
|
||||
tool
|
||||
for tool in conversation.tools
|
||||
if not tool.defer_loading or tool.name in loaded
|
||||
]
|
||||
|
||||
|
||||
def _error_message(body: Any) -> str:
|
||||
try:
|
||||
return json.loads(body)["error"]["message"]
|
||||
@@ -110,6 +130,20 @@ class DeepSeekRecipe:
|
||||
recipe = load_deepseek_recipe()
|
||||
self.model = MODEL_NAMES[chat_encoding_spec]
|
||||
self.hf_tokenizer = tokenizer
|
||||
self._fast_prompt_tokenizer = None
|
||||
if type(tokenizer).__module__.partition(".")[0] == "fastokens":
|
||||
from fastokens import Tokenizer
|
||||
|
||||
if hasattr(Tokenizer, "encode_segments"):
|
||||
self._fast_prompt_tokenizer = Tokenizer.from_json_str(
|
||||
tokenizer.to_str()
|
||||
)
|
||||
logger.info("DeepSeek recipe prompt encoder: fastokens segmented")
|
||||
else:
|
||||
logger.warning(
|
||||
"deepseek-recipe prompt acceleration requires fastokens.encode_segments; "
|
||||
"using the reference prompt tokenizer"
|
||||
)
|
||||
self.special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
|
||||
self._options = recipe.ConversionOptions().with_special_tokens(
|
||||
self.special_tokens
|
||||
@@ -146,6 +180,19 @@ class DeepSeekRecipe:
|
||||
) -> list[int]:
|
||||
"""Prompt ids for `body` (see `pre_normalize`); a request recipe rejects
|
||||
raises ValueError. `image_urls` receives the images in placeholder order."""
|
||||
conversation = self.convert(body, effort_budget=effort_budget)
|
||||
prompt_ids = self.encode_conversation(conversation, effort_budget=effort_budget)
|
||||
if image_urls is not None:
|
||||
image_urls.extend(self.image_urls(conversation))
|
||||
if task is not None:
|
||||
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
|
||||
return prompt_ids
|
||||
|
||||
def convert(
|
||||
self, body: dict[str, Any], *, effort_budget: Optional[int] = None
|
||||
) -> Any:
|
||||
"""Convert `body` to recipe's `Conversation`; invalid requests raise ValueError.
|
||||
Pass the same `effort_budget` to `encode_conversation`."""
|
||||
recipe = load_deepseek_recipe()
|
||||
normalized = pre_normalize(body)
|
||||
normalized["model"] = self.model
|
||||
@@ -155,16 +202,104 @@ class DeepSeekRecipe:
|
||||
converted = recipe.ChatCompletionRequest(normalized).convert(self._options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
prompt_ids = self._encoding.encode(converted.conversation)
|
||||
if image_urls is not None:
|
||||
rendered = self._encoding.render_conversation(converted.conversation)
|
||||
image_urls.extend(
|
||||
source.url or source.data_url for source in rendered.image_sources
|
||||
)
|
||||
if effort_budget is not None and body["thinking"]:
|
||||
return converted.conversation
|
||||
|
||||
def convert_native(
|
||||
self, protocol: str, body: dict[str, Any], *, thinking: bool
|
||||
) -> Any:
|
||||
"""Convert a Responses or Messages request to recipe's `Conversation` in the
|
||||
host's thinking mode; invalid requests raise ValueError."""
|
||||
recipe = load_deepseek_recipe()
|
||||
request_type = {
|
||||
"responses": recipe.ResponsesRequest,
|
||||
"messages": recipe.MessagesRequest,
|
||||
}[protocol]
|
||||
options = self._options.with_default_thinking_mode(thinking)
|
||||
try:
|
||||
converted = request_type({**body, "model": self.model}).convert(options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
return converted.conversation
|
||||
|
||||
def with_host_settings(
|
||||
self,
|
||||
conversation: Any,
|
||||
*,
|
||||
response_schema: Optional[dict[str, Any]],
|
||||
continue_final_message: bool,
|
||||
) -> Any:
|
||||
"""`conversation` with what recipe's Responses and Messages conversions
|
||||
leave out: a JSON response schema, and SGLang's reading of a trailing
|
||||
assistant turn, continued or else re-sent as a user turn."""
|
||||
recipe = load_deepseek_recipe()
|
||||
messages = conversation.messages
|
||||
last = messages[-1] if messages else None
|
||||
if last is not None and last.role == "assistant":
|
||||
# recipe closes the turn and opens another; SGLang continues it or,
|
||||
# as `_handle_last_assistant_message`, re-sends it as a user turn.
|
||||
if continue_final_message:
|
||||
last = recipe.AssistantMessage(
|
||||
last.content, last.reasoning_content, prefix=True
|
||||
)
|
||||
else:
|
||||
last = recipe.UserMessage(last.content)
|
||||
messages = [*messages[:-1], last]
|
||||
elif response_schema is None:
|
||||
return conversation
|
||||
return recipe.Conversation(
|
||||
messages,
|
||||
thinking_mode=conversation.thinking_mode,
|
||||
tools=conversation.tools,
|
||||
tool_choice=conversation.tool_choice,
|
||||
reasoning_effort=conversation.reasoning_effort,
|
||||
response_schema=(
|
||||
response_schema
|
||||
if response_schema is not None
|
||||
else conversation.response_schema
|
||||
),
|
||||
)
|
||||
|
||||
def namespace_tool_names(
|
||||
self, namespaces: list[dict[str, Any]]
|
||||
) -> dict[str, tuple[str, str]]:
|
||||
"""The name recipe's Responses conversion calls each member of `namespaces`
|
||||
by, mapped to its namespace and its name there. `namespaces` lists the
|
||||
request's namespace tools in conversion order: declared, then discovered."""
|
||||
recipe = load_deepseek_recipe()
|
||||
body = {"model": self.model, "input": "", "tools": namespaces}
|
||||
try:
|
||||
return recipe.ResponsesRequest(body).namespace_tool_map(self._options)
|
||||
except recipe.ConversionError as error:
|
||||
raise ValueError(_error_message(error.body)) from error
|
||||
|
||||
def image_urls(self, conversation: Any) -> list[str]:
|
||||
"""The images of `conversation` in the order its prompt places them."""
|
||||
rendered = self._encoding.render_conversation(conversation)
|
||||
return [source.url or source.data_url for source in rendered.image_sources]
|
||||
|
||||
def encode_conversation(
|
||||
self, conversation: Any, *, effort_budget: Optional[int] = None
|
||||
) -> list[int]:
|
||||
"""Encode a recipe `Conversation`, including a prefix of its messages."""
|
||||
if self._fast_prompt_tokenizer is None:
|
||||
prompt_ids = self._encoding.encode(conversation)
|
||||
else:
|
||||
rendered = self._encoding.render_conversation(conversation)
|
||||
prompt = rendered.prompt
|
||||
segments = []
|
||||
cursor = 0
|
||||
# Recipe distinguishes control tokens from identical spellings in
|
||||
# user text. Its Python ranges use code-point offsets, not bytes.
|
||||
for start, end in rendered.special_ranges:
|
||||
if start > cursor:
|
||||
segments.append((prompt[cursor:start], False))
|
||||
segments.append((prompt[start:end], True))
|
||||
cursor = end
|
||||
if cursor < len(prompt):
|
||||
segments.append((prompt[cursor:], False))
|
||||
prompt_ids = self._fast_prompt_tokenizer.encode_segments(segments).ids
|
||||
if effort_budget is not None and conversation.thinking_mode:
|
||||
prompt_ids = self._replace_effort_budget(prompt_ids, effort_budget)
|
||||
if task is not None:
|
||||
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
|
||||
return prompt_ids
|
||||
|
||||
def parse_start(
|
||||
|
||||
@@ -97,7 +97,9 @@ class AnthropicMessagesMixin:
|
||||
)
|
||||
|
||||
serving = AnthropicServing(
|
||||
openai_serving_chat=SimpleNamespace(supports_inline_system=True)
|
||||
openai_serving_chat=SimpleNamespace(
|
||||
supports_inline_system=True, native_tool_loading=False
|
||||
)
|
||||
)
|
||||
chat_request = serving._convert_to_chat_completion_request(anthropic_request)
|
||||
converted = chat_request.model_dump()
|
||||
|
||||
@@ -31,6 +31,7 @@ register_cpu_ci(est_time=11, suite="base-a-test-cpu")
|
||||
|
||||
class _FakeOpenAIServingChat:
|
||||
native_reasoning_history = False
|
||||
native_tool_loading = False
|
||||
|
||||
def __init__(self, stream_lines=None, chat_template=None):
|
||||
self.stream_lines = stream_lines or []
|
||||
@@ -61,6 +62,7 @@ class _FakeNonStreamingErrorOpenAI:
|
||||
"""Returns an error response or raises an exception from the OpenAI handler."""
|
||||
|
||||
supports_inline_system = False
|
||||
native_tool_loading = False
|
||||
|
||||
def __init__(self, status_code=400, body=None, content=None, exception=None):
|
||||
self._status_code = status_code
|
||||
@@ -496,6 +498,26 @@ class TestAnthropicServing(unittest.TestCase):
|
||||
],
|
||||
)
|
||||
|
||||
def test_tool_addition_fails_where_tools_cannot_load_mid_conversation(self):
|
||||
"""A tool_addition block must not vanish on a model that cannot render it."""
|
||||
request = self._anthropic_request(
|
||||
messages=[
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "system",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_addition",
|
||||
"tool": {"type": "tool_reference", "name": "Bash"},
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "tool_addition"):
|
||||
self._serving()._convert_to_chat_completion_request(request)
|
||||
|
||||
def test_builtin_web_search_tool_without_schema_is_skipped(self):
|
||||
request = AnthropicMessagesRequest.model_validate(
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -204,6 +204,25 @@ class TestDeepSeekRecipe(CustomTestCase):
|
||||
}
|
||||
)
|
||||
|
||||
def test_fastokens_preserves_prompt_ids(self):
|
||||
try:
|
||||
from fastokens import Tokenizer, _ConfiguredTokenizerShim
|
||||
except ImportError:
|
||||
self.skipTest("fastokens is not installed")
|
||||
if not hasattr(Tokenizer, "encode_segments"):
|
||||
self.skipTest("fastokens has no segmented encoder")
|
||||
candidate = renderer.DeepSeekRecipe(
|
||||
"dsv41", _ConfiguredTokenizerShim(self.tokenizer)
|
||||
)
|
||||
self.assertIsNotNone(candidate._fast_prompt_tokenizer)
|
||||
message = {"role": "user", "content": "中文 👩🏽💻 <think>x</think>"}
|
||||
for thinking in (False, True):
|
||||
body = {"messages": [message], "thinking": thinking}
|
||||
with self.subTest(thinking=thinking):
|
||||
self.assertEqual(
|
||||
candidate.encode(body), self.deepseek_recipe.encode(body)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user