Compare commits

...
Author SHA1 Message Date
minke.yu 8b67636010 Merge b300/integration into main: 2026-11-17 c580 baseline
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included

This is the base for the next Dynamo layer.
2026-11-17 20:08:08 +08:00
minke.yu 310ad35430 deploy: record 2026-11-17 c580 baseline (D16+fastokens) and experiments 2026-11-17 20:05:11 +08:00
minke.yu bdbd8dffb8 deploy(ds41-gist-exp): P buffer_only + optimistic, D tokenizer 16 2026-11-16 22:03:20 +08:00
minke.yu f363a6ab7b deploy: record 11/16 pre-optimistic stable config 2026-11-16 22:03:11 +08:00
minke.yu e4c7e540be deploy(g9i): T26 qps20 + P queue32 production candidate 2026-11-16 00:42:44 +08:00
minke.yu b8b205b367 deploy(g9i): merged ns+td gateway, 32-request P queue cap (p95 5.70s) 2026-11-15 22:25:33 +08:00
minke.yu b3d6d19c7d deploy(g9i): set regular P queue reject 32s (experiment) 2026-11-15 21:45:51 +08:00
minke.yu 8f53c1fb73 deploy(l3ssd): offload_on_evict=false, 4TB cap, L3 reset script 2026-11-15 20:33:38 +08:00
minke.yu 42f228bfb8 docs(deploy): fresh-instance multi-round tokenizer A/B (B2 TTFT p95 < 5s) 2026-11-15 18:20:53 +08:00
minke.yu 242a7ad23a docs(deploy): record L3 SSD tokenizer A/B (fastokens vs huggingface) 2026-11-15 15:49:56 +08:00
minke.yu 2eb163ad73 fix(deploy): point sglang cache at the shared warm cache 2026-11-15 14:23:52 +08:00
minke.yu d3240c745e deploy(b300): L3 Mooncake SSD baseline with mooncake 0.3.14 2026-11-15 14:10:30 +08:00
minke.yu 433257c7b0 fix(recipe): drop the pre-48928 _fast_prompt_tokenizer fallback stub 2026-11-15 10:53:15 +08:00
leon 7980a09c0a [Perf] DeepSeek-V4.1-Flash: Use segmented fastokens for prompt encoding (#48928)
(cherry picked from commit 219b7e7a688363bac9ef09eacf80e44eef9efe4b)
2026-11-15 10:51:40 +08:00
minke.yu a7c668a936 fix(chat): move namespace-tool flattening validator into ChatCompletionRequest 2026-11-14 14:10:07 +08:00
minke.yu 0c6849e378 chat: flatten namespace tools to ns__function for Chat Completions 2026-11-14 14:09:36 +08:00
minke.yu 85605b8c72 fix(protocol): ResponseTool accepts extra fields (tool_search execution) 2026-11-13 23:25:50 +08:00
minke.yu b6a5d70cba fix(responses): tolerate tools without model_extra in native validation 2026-11-13 23:24:00 +08:00
minke.yu cac60f7c2e fix(protocol): effective_tool_choice preserves namespace/type 2026-11-13 23:22:21 +08:00
minke.yu 68c2af7957 fix(protocol): Function.strict is Optional[bool] for recipe tool schemas 2026-11-13 23:21:04 +08:00
minke.yu 497aeff5f6 fix(recipe): define _fast_prompt_tokenizer fallback on this base 2026-11-13 22:44:35 +08:00
minke.yu 5fa8c23227 vendor: DeepSeek-recipe namespace tools (#46370 + #49625)
Backport the upstream namespace-tool stack onto the B300 recipe base:
- #46370: Function.namespace / ChatMessage.action protocol fields.
- #49625: Responses/Messages native recipe rendering (convert_native),
  tool_search/defer_loading, namespace identity in/out/history replay.

Adapted to this base: skips serving_kimi_k3.py (not present), drops the
Minimax/Beam/wire-only dependencies added between 10-23 and the PR base,
and keeps the existing tool-parsing fallback where the newer
`parses_without_tools()` helper is unavailable.

Also includes the PR's Anthropic tool-reference plumbing and unit tests.
2026-11-13 22:43:37 +08:00
45 changed files with 12930 additions and 97 deletions
@@ -0,0 +1,25 @@
# L3 SSD image: recipe-ns engine + official mooncake 0.3.14 (CUDA 13, cp312).
#
# Build context: this directory plus the 0.3.14 wheel
# mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# Use build-l3ssd-image.sh, which downloads and verifies the wheel.
#
# Base tag recipe-ns-433257c7b-baked-20261114 = b300/integration @ 433257c7b
# (recipe namespace/chat-ns + gateway binary baked). Rebuild it from that commit
# if the local tag is missing.
FROM sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-433257c7b-baked-20261114
COPY mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/
RUN python3 -m pip install --no-deps --force-reinstall \
/tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
&& rm -f /tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
# The wheel's master binary uses $ORIGIN/../mooncake_transfer_engine_cuda13.libs;
# keep it in place and expose a wrapper at the legacy path used by the manifests.
&& printf '%s\n' \
'#!/bin/sh' \
'export LD_LIBRARY_PATH=/opt/sglang/lib/python3.12/site-packages/mooncake_transfer_engine_cuda13.libs:$LD_LIBRARY_PATH' \
'exec /opt/sglang/lib/python3.12/site-packages/mooncake/mooncake_master "$@"' \
> /opt/sglang/bin/mooncake_master \
&& chmod +x /opt/sglang/bin/mooncake_master \
&& python3 -c "import importlib.metadata as m; print('mooncake:', m.version('mooncake-transfer-engine-cuda13'))" \
&& /opt/sglang/bin/mooncake_master --version | head -1
+139
View File
@@ -0,0 +1,139 @@
# B300 L3 Mooncake SSD 基线部署(2026-11-15)
> 以同事在 `ds41-gist-exp` 上验证的 **L3 Mooncake SSD** 配置为基础,使用我们的
> `b300/integration` 引擎代码 + 官方 **mooncake 0.3.14**,不再依赖 hostPath
> `/opt/mooncake-latest`。
## 1. 版本要求(重要)
- **mooncake `0.3.14`**(`mooncake-transfer-engine-cuda13`,CUDA 13 / cp312 wheel)。
- 镜像内原为 `0.3.13.post1`;0.3.14 含上游 SSD offload 相关修复/增强。
- wheel 与 sha256 见 `requirements-mooncake.txt`,安装必须 `--no-deps`(基底已有运行时依赖)。
- 引擎:`b300/integration` @ `433257c7b`(recipe namespace/chat-ns + 新 gateway)。
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
(40.1GB,已 `ctr -n k8s.io images import`;本地 docker tag `ymkymx/sglang:...` 同名)。
- 构建/装载:`./build-l3ssd-image.sh`(下载校验 wheel → docker build → ctr import)。
镜像内 `/opt/sglang/bin/mooncake_master` 是 wrapper:设置 wheel `.libs` 的
`LD_LIBRARY_PATH` 后 exec 包内 master(已验证 `0.3.14 (git e1ea5d3)`)。
## 2. L3 SSD 配置
`mooncake-store-l3ssd.json`:
```json
{"master_server_address":"127.0.0.1:50051","metadata_server":"P2PHANDSHAKE",
"global_segment_size":"200gb","protocol":"tcp",
"enable_ssd_offload":true,"ssd_offload_path":"/data/ymk/mooncake-ssd"}
```
- 部署时把该文件放到节点 `/data/ymk/ds41/mooncake-store-l3ssd.json`,SSD 目录
`/data/ymk/mooncake-ssd` 需存在。
- `MOONCAKE_OFFLOAD_*`:`bucket_storage_backend` / 512GiB bucket 上限 / `fifo` /
`flush_partial=0`(见 manifest)。
- master 参数必须带
`--enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false`。
## 3. manifest
| 文件 | 用途 |
|---|---|
| `ds41-l3ssd-test-baked-20261115.yaml` | 新建 cw `ds41-l3ssd-test`(推荐先做基线验证) |
| `ds41-gist-exp-l3ssd-baked-20261115.yaml` | 原名 `ds41-gist-exp`,用于直接更新原栈 |
已通过 `kmctl create/update --dry-run` 校验。启动前先停掉占用 8 卡的 stack
(`ds41-recipe-ns-test` 或旧 `ds41-gist-exp`),确认 GPU 归零:
```bash
kmctl create cw ds41-l3ssd-test --node-group B300 \
-f deploy/b300-k8s/ds41-l3ssd-test-baked-20261115.yaml
# 或替换原栈:
# kmctl update cw ds41-gist-exp -f deploy/b300-k8s/ds41-gist-exp-l3ssd-baked-20261115.yaml
```
端口/角色与旧 `ds41-gist-exp` 一致:master 50051、P 30000、D 30300、gateway 30400。
启动约 10-15 分钟;核对 P/D `max_total_num_tokens` 和 master 日志中的 SSD offload 初始化。
## 4. 与上一版配置的差异
P/D 的 `sglang serve` 参数**完全一致**;差异只在 mooncake:
| 项 | 旧(11-09) | 本基线 |
|---|---|---|
| mooncake | 镜像内 0.3.13.post1 | 镜像内 **0.3.14** |
| master | `/opt/sglang/bin/mooncake_master`,无 offload | 同路径(0.3.14 wrapper),带 offload 参数 |
| P/D PYTHONPATH | 无 | 无(0.3.14 已烘入) |
| L3 配置 | 纯 TCP,无 SSD | `enable_ssd_offload` + `ssd_offload_path` |
| `MOONCAKE_OFFLOAD_*` | 无 | bucket backend / 512GiB / fifo / flush_partial=0 |
| TMPDIR/XDG/HF | 默认/`/data/ymk` | `/data/ymk/l3ssd/{master,p,d}-*` |
## 5. 注意
- 同事原配置使用 `SGLANG_TRACE_LOOP_SAMPLE=1.0`(全量 detail tracing,约 2% 吞吐 +
p95 开销)。本基线为对齐其配置保留 1.0;正式上线建议改回 `0.05`。
- `global_segment_size=200gb` + 512GiB SSD offload:上线前按宿主内存总账复核。
- SSD 目录增长需监控(`/data` 当前约 7.6T 可用)。
- 旧镜像 `recipe-ns-433257c7b-baked-20261114` 与 `recipe-ns-a7c668a93-baked-20261114`
保留作回滚/对照。
## 6. Tokenizer A/B(2026-11-15,c580 / 1200s)
同一 `ds41-gist-exp`(L3 SSD + mooncake 0.3.14)整栈重启,只切 P/D
`--tokenizer-backend`;AgentX c580 / 1200s / warmup 256 / seed 42,经 30400。
| 臂 | tokenizer | rps | in tok/s | out tok/s | TTFT p50/p95/p99 | ITL p50/p95/p99 | requests |
|---|---|---:|---:|---:|---|---|---:|
| A | `huggingface`(默认) | 23.789 | 2,257,661 | 22,224 | 2.043 / 4.176 / 7.579 s | 8.02 / 14.48 / 20.91 ms | 29,498 |
| B | **`fastokens`** | **24.045** | **2,289,885** | **22,475** | **1.746 / 3.676 / 6.820 s** | 8.13 / 11.65 / 14.87 ms | 29,816 |
- 结论:fastokens 臂 **rps +1.1%**、TTFT p50/p95/p99 **−14.5% / −12.0% / −10.0%**;
ITL p95 的差异大概率是 run-to-run 噪声(decode 不走 tokenizer),不作为判据。
- #48928 的分段编码已生效:容器内 `DeepSeekRecipe.from_hf_tokenizer(..., fastokens tokenizer)`
的 `_fast_prompt_tokenizer` 非 None(`fastokens.Tokenizer`);服务端未打印
“fastokens segmented” 日志,属日志路径差异,不影响结论。
- **当前线上实验栈保留 fastokens 臂**;要回默认用
`ds41-gist-exp-l3ssd-baked-20261115.yaml` 更新并整栈重启。
- 单轮对比,c580 本身有方差;如需正式定论建议每臂 ≥3 轮。
- SSD offload 已配置(master 报 `SSD Storage: 0 B / 2.00 TB`),本次 c580 工作集未触发
SSD 逐出(`/data/ymk/mooncake-ssd` 约 20K),不是配置未生效。
## 7. 多轮验证(2026-11-15,每轮整栈重启)
> 之前 B2/B3 在同一实例上连跑、未重启,数据受实例状态影响;下面只列**整栈重启后**的轮次。
> 整栈重启 = `kmctl stop` + `update`(换臂),或同臂时删除 p/d/gateway/master 四个 Pod。
| 臂 | 轮次 | rps | TTFT p50/p95/p99 (s) | ITL p95 (ms) | requests |
|---|---|---:|---|---:|---:|
| A(huggingface) | A1 | 23.789 | 2.043 / 4.176 / 7.579 | 14.48 | 29,498 |
| A | A2 | 23.552 | 2.305 / 5.894 / 8.600 | 15.05 | 29,205 |
| B(fastokens) | B1 | 24.045 | 1.746 / 3.676 / 6.820 | 11.65 | 29,816 |
| B | **B2(fresh)** | **24.040** | **1.708 / 3.451 / 7.105** | 11.71 | 29,810 |
- **B2 满足判据:TTFT p95 = 3.451s < 5s**,且与 B1(3.676s)一致;
A 臂 fresh 轮 p95 为 4.176 / 5.894s。
- fresh 轮均值:A rps 23.67 / p95 5.04s;B rps 24.04 / p95 3.56s
(B +1.6% rps、p95 −29%)。
- 结论:保留 fastokens;之前未重启的 B2/B3(p95 5.99/6.69s)仅作历史参考、不纳入。
## 8. L3 SSD 4TB + offload_on_evict=false(2026-11-15)
- 启动清单 `ds41-gist-exp-l3ssd-fastokens-20261115.yaml` 已改:master
`--offload_on_evict=false`;P 的 `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` /
`MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` 改为 1 TiB/rank,4 rank 合计
master 报 `SSD Storage: 0 B / 4.00 TB`。
- 每次实验前必须重置 L3:stop cw → 确认 Terminated 0/0 + GPU 0 MiB →
`bash clean_l3ssd_cache.sh`(删除 `/data/ymk/mooncake-ssd` 下的 rank_*)→ update/start。
本次清理释放 1.2T,`/data` 可用 6.4T→7.5T。
- c580(1200s,warmup 256,seed 42,fastokens,干净 L3):
rps 24.003、in 2.285M tok/s、out 22,447、TTFT p50/p95/p99 1.768/3.500/7.549s、
ITL 8.15/11.74/15.31ms、29,764 请求、0 错误、valid。
对比 B1/B2:24.045/1.746/3.676/6.820 与 24.040/1.708/3.451/7.105——
吞吐与尾延迟均在方差内,**没有可归因于 offload_on_evict/4TB/清理的收益或回归**。
- **重要:`offload_on_evict=false` 并没有停止 SSD 写入**。run 后 master
`SSD Storage: 533.81 GB / 4.00 TB`,`du /data/ymk/mooncake-ssd = 535G`;
这是 `--hicache-write-policy write_through` 的 backup 写入,和内存 eviction 无关。
若要真正避免 SSD 写入,需要改 `write_back` 或关闭 `--enable_offload`,但那会改变
L3 语义,需另做 A/B。
- run 后 P cgroup `memory.current=2,198,904,565,760` ≈ 2047.5 GiB,已顶到 2 TiB
limit;宿主 MemAvailable 1.2 TiB。下一次实验前必须先停/清/重启;不要在同一实例
上连续跑,否则 L3 与 cgroup 都处于顶格状态。
@@ -0,0 +1,10 @@
# 2026-11-17 c580 基线:D tokenizer 16 + fastokens
- manifest: `manifest.yaml`(= `ds41-gist-exp-20261117-dtok16-fastokens.yaml`)
- P: cache / 无 optimistic / tokenizer-worker 16 / fastokens / hicache ratio 4 / L3 SSD 4T
- D: tokenizer-worker 16 / fastokens
- Gateway: 正常(无 P admission cap,health/circuit 开启,decode-max-running 512)
- L3:每轮前清空;完整重启
- c580 结果:rps 24.045,in 2.290M tok/s,out 22.449k tok/s,
TTFT 1.743/3.772/6.580 s,ITL p95 10.74 ms,29,816 req,valid=true。
- 原始结果:`experiments/agentx-align/results-c580-dtok16-fastokens-20261117/`
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1 @@
{"rps": 24.04512015569562, "in_tput": 2290117.7612668574, "out_tput": 22449.32127276801, "ttft_p50": 1743.335193, "ttft_p95": 3771.88552025, "ttft_p99": 6579.86958935, "itl_p50": 8.162268671611512, "itl_p95": 10.741317697249674, "itl_p99": 11.806237803568294, "lat_p95": 31884.86991075, "requests": 29816.0, "isl_p50": 79909.0, "osl_p50": 339.0, "cache_pct": NaN, "valid": true}
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Build and load the L3 SSD image (mooncake 0.3.14) on b300-01.
set -euo pipefail
cd "$(dirname "$0")"
WHEEL=mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
URL=https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/$WHEEL
SHA=74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
TAG=recipe-ns-l3ssd-433257c7b-baked-20261115
REPO=sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang
if [ ! -f "$WHEEL" ]; then
curl -fL -o "$WHEEL" "$URL"
fi
echo "$SHA $WHEEL" | sha256sum -c -
docker build -t "ymkymx/sglang:$TAG" -f Dockerfile.mooncake-0.3.14 .
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
echo "loaded $REPO:$TAG"
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Reset the mooncake L3 SSD KV cache before an experiment.
#
# MUST run on b300-01 only after the ds41-gist-exp workload is fully stopped:
# 1) kmctl stop cw ds41-gist-exp --wait ...
# 2) kubectl get cw ... -> Terminated 0/0, nvidia-smi -> 0 MiB on GPU0-7
# 3) bash clean_l3ssd_cache.sh
# 4) kmctl update/start the new manifest
#
# The script refuses to delete anything while a mooncake master or sglang
# server process is still visible on the host.
set -euo pipefail
SSD_DIR="${SSD_DIR:-/data/ymk/mooncake-ssd}"
if pgrep -af '[m]ooncake_master|[s]glang serve' >/dev/null 2>&1; then
echo "refusing: mooncake_master or sglang serve is still running" >&2
pgrep -af '[m]ooncake_master|[s]glang serve' >&2 || true
exit 1
fi
if [ ! -d "$SSD_DIR" ]; then
echo "SSD cache dir does not exist: $SSD_DIR"
exit 0
fi
echo "before: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
find "$SSD_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
echo "after: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
df -h /data | tail -1
@@ -0,0 +1,542 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 32 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n\
\ --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30\
\ --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\
\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 --g9i-p-qps 20 --g9i-p-refresh-ms 250 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-l3ssd-test
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,62 @@
<h1>当前基线:ds41-gist-exp(4T L3 SSD,2026-11-17)</h1>
<callout emoji="📌" background-color="light-blue"><p>本节是 2026-11-17 定稿的 c580/c620 基线,后续 Dynamo 叠加以此为底座。旧的 ds41-gist-1p1d 章节已归档,不再作为当前部署。</p></callout>
<h2>1. 基线身份与镜像</h2>
<ul>
<li><b>工作负载</b>:<code>ds41-gist-exp</code> / <code>cw-dblft6sgrhagmpdt8ef0</code>,namespace <code>prj-d7te67bj0sq2fgdrmbig</code>,b300-01。</li>
<li><b>镜像</b>:<code>sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115</code>(40.1GB,含 Mooncake 0.3.14 + L3 SSD 支持)。</li>
<li><b>manifest</b>:<code>deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml</code>;基线副本 <code>deploy/b300-k8s/baselines/20261117-dtok16-fastokens/manifest.yaml</code>;b300 副本在 <code>/data/ymk/deploy/b300-k8s/</code>。</li>
<li><b>形态</b>:RBG 三角色 <code>p</code> / <code>d</code> / <code>gateway</code> + <code>mooncake-master</code>;全 hostNetwork / hostPID / privileged;P=GPU0-3,D=GPU4-7。</li>
</ul>
<h2>2. 部署方式与端口</h2>
<table>
<thead><tr><th><p>角色</p></th><th><p>端口</p></th><th><p>说明</p></th></tr></thead>
<tbody>
<tr><td><p>P</p></td><td><p>30000 / 8998 / 9400-9403</p></td><td><p>HTTP / bootstrap / Mooncake client metrics</p></td></tr>
<tr><td><p>D</p></td><td><p>30300</p></td><td><p>HTTP</p></td></tr>
<tr><td><p>gateway</p></td><td><p>30400 / 29000</p></td><td><p>业务 / metrics</p></td></tr>
<tr><td><p>mooncake-master</p></td><td><p>50051 / 18081 / 19003</p></td><td><p>store / metadata / metrics</p></td></tr>
</tbody>
</table>
<p>默认入口:<code>http://127.0.0.1:30400</code>;当前 new-api/AgentX 直接使用该入口。</p>
<h2>3. 关键参数</h2>
<p><b>P(prefill)</b>:tp4 / dp4 / ep4,megamoe + flashmla_mega,ep-redundant 32,DSpark block 3,decoder SWA bounded replay;chunked-prefill 131072 / max-prefill 131072,max-running 512;fastokens + tokenizer-worker 16;HiCache ratio 4、page_first_direct、direct、write_through、Mooncake L3;<b>cache 模式,无 buffer_only、无 optimistic</b>;mem-fraction 0.80。</p>
<p><b>D(decode)</b>:tp4 / dp4 / ep4,ep-redundant 64,DSpark block 3,stream-interval 32,swa-prefix-tails 2048;max-running 512,cuda-graph-max-bs-decode 512;chunked 32768 / max-prefill 8192;fastokens + tokenizer-worker 16;mem-fraction 0.80。</p>
<p><b>gateway</b>:镜像内 <code>sgl-model-gateway</code>;<b>无 P admission cap</b>(无 p-qps / p-queue-reqs-cap / p-queue-token-cap,td-admit 默认 off);保留 health check + circuit breaker;<code>--g9i-decode-max-running 512</code>。这是 c580 能跑通的关键。</p>
<p><b>L3</b>:Mooncake master 4TB SSD offload;<code>offload_on_evict=false</code>;L3 配置见 <code>/data/ymk/ds41/mooncake-store-l3ssd.json</code>。</p>
<h2>4. 启动 / 重启流程</h2>
<pre lang="bash"><code># 1) 停完整工作负载并确认 GPU 0
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 "nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
# 2) 清 L3 SSD(实验/基线前必须)
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
# 3) 用完整平台 manifest 启动(不要只 kubectl patch gateway)
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml
# 4) 等 4 Pod 1/1,检查 P/D/gateway health 200,再 smoke
curl -sS http://127.0.0.1:30400/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"/data/models/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"1+1=?"}],"max_tokens":8}'
</code></pre>
<p>启动基线:P <code>max_total_num_tokens≈16,158,208</code> / 128 per-rank / avail ≈23GB;D <code>58,859,520</code> / 128 per-rank / avail ≈79GB。正常 warmup 6,385 条约 6.5 分钟。</p>
<h2>5. 基线结果</h2>
<table>
<thead><tr><th><p>场景</p></th><th><p>RPS</p></th><th><p>TTFT p50/p95/p99</p></th><th><p>ITL p95</p></th><th><p>结论</p></th></tr></thead>
<tbody>
<tr><td><p>c580 基线</p></td><td><p>24.045</p></td><td><p>1.743 / 3.772 / 6.580 s</p></td><td><p>10.74 ms</p></td><td><p>推荐工作点</p></td></tr>
<tr><td><p>c620 首轮(清 L3)</p></td><td><p>25.998</p></td><td><p>1.814 / 4.041 / 8.257 s</p></td><td><p>11.69 ms</p></td><td><p>吞吐 +8%,尾部变差</p></td></tr>
<tr><td><p>c620 不重启 r2/r3</p></td><td><p>25.85 / 25.61</p></td><td><p>p95 5.39 / 6.44 s,p99 9.56 / 10.79 s</p></td><td><p>10.96 / 10.68 ms</p></td><td><p>D KV 接近/打满,TTFT 尾部持续劣化,不建议连测</p></td></tr>
<tr><td><p>buffer_only</p></td><td><p>23.749</p></td><td><p>2.153 / 4.904 / 10.483 s</p></td><td><p>10.32 ms</p></td><td><p>吞吐 -1.2%、TTFT 更差,不采用</p></td></tr>
</tbody>
</table>
<p>结论:<b>c580 是稳定基线</b>;c620 只适合短时冲吞吐,且每轮之间必须重启/清池。buffer_only 不作为基线。</p>
<h2>6. 作为 Dynamo 底座</h2>
<p>当前镜像是 <b>SGLang + Mooncake 0.3.14 + L3 SSD</b> 基线,<b>不包含 Dynamo/nixl</b>。后续套 Dynamo 需要以本镜像为 base 另装 Dynamo 1.6.x / nixl,旧 <code>gist-nvlink-7f6fe2244-dynamo-local-20261026</code> 是更早的 10-23 代码,不是本基线。</p>
<p>Dynamo 叠加时保留本页的 P/D 参数、L3 SSD 和 gateway 入口;先做单机 4P4D 冒烟,再评估 Dynamo 的 P/D 角色、discovery 和 KV 传输配置。</p>
@@ -0,0 +1,8 @@
{
"master_server_address": "127.0.0.1:50051",
"metadata_server": "P2PHANDSHAKE",
"global_segment_size": "200gb",
"protocol": "tcp",
"enable_ssd_offload": true,
"ssd_offload_path": "/data/ymk/mooncake-ssd"
}
@@ -0,0 +1,6 @@
# L3 Mooncake SSD baseline requires the official 0.3.14 wheel.
# Previous image shipped 0.3.13.post1; 0.3.14 contains the upstream SSD-offload fixes.
# Wheel: https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# sha256: 74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
# Installed with --no-deps; the base image already provides the runtime deps.
mooncake-transfer-engine-cuda13==0.3.14
@@ -0,0 +1,44 @@
# 2026-11-16 buffer_only + optimistic 首启记录
## 变更(相对 `stable-20261116-pre-optimistic`)
P:
- 增加 `--hicache-host-memory-mode buffer_only`(保留 `--hicache-write-policy write_through`、
`--hicache-ratio 4`、mooncake L3)
- 增加 `--optimistic-prefill-attempts 4`
D:
- 增加 `--tokenizer-worker-num 16`(与 P 相同)
其余角色/参数不变;网关保持 live 的 `p-queue-reqs-cap 64 / token-cap 32M / p-qps 40`。
## 启动前清理
- `kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s` → `Terminated 0/0`,GPU 0-7 全 0 MiB
- `/data/ymk/mooncake-ssd`:2.3T → 4.0K;`/data` 7.5T available
- `kmctl update cw ds41-gist-exp -f ds41-gist-exp-20261116-bufferonly-optimistic.yaml`
## 首启验证(2026-11-16 22:00 CST)
- 4 个 Pod 全 `1/1 Running`;P/D/gateway `/health` 全 200
- P `server_args`:`hicache_host_memory_mode='buffer_only'`、`optimistic_prefill_attempts=4`;
没有 "Optimistic prefill supports ..." 的禁用 warning
- D `server_args`:`tokenizer_worker_num=16`;日志有 16 个 multi-tokenizer worker
- 启动基线:P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`
- 冒烟:chat `/v1/chat/completions` 200,回答正确;16 并发 4k 字符请求全 200
- **optimistic 已生效**:P 日志出现 `#optimistic-req: 1`(其余为 0)
- P/D 日志无 Traceback / ERROR / assert
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/sglang/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
代码仓库标签:`stable-20261116-pre-optimistic`。
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T13:54:16Z"
generation: 1
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30768532"
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T14:01:52Z"
message: All roles are ready
observedGeneration: 1
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 1
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,68 @@
# 11/16 上线前稳定配置(ds41-gist-exp)
记录时间:2026-11-16 21:49 CST(13:49Z),在切换 `buffer_only + optimistic` 之前。
## 身份
- 平台 cw:`ds41-gist-exp` / `cw-dblft6sgrhagmpdt8ef0`
- RBG:`ds41-gist-exp`(namespace `prj-d7te67bj0sq2fgdrmbig`)
- 角色:`mooncake-master` / `p` / `d` / `gateway`
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
- P:GPU 0-3;D:GPU 4-7;全 hostNetwork / hostPID / privileged,钉 b300-01
## 文件
- `rbg-live-raw.yaml`:`kubectl get rbg -o yaml` 原始快照(含 status,仅存档)
- `cw-ds41-gist-exp-11-16-pre-optimistic.yaml`:清理过平台字段的 RBG,可直接用于回滚更新
- `cw-platform.yaml`:`kmctl get cw -o yaml` 的平台清单(`Spec.Manifest` 是旧值,见下方漂移)
- 本说明
## 生效配置摘要
P:
- `tp4 / dp4 / ep4`、`megamoe`、`flashmla_mega`、`ep-redundant 32`
- DSPARK block 3、decoder SWA bounded replay、`swa-full-tokens-ratio 0.12`
- `chunked-prefill-size 131072`、`max-prefill-tokens 131072`
- `max-running-requests 512`、`tokenizer-worker-num 16`
- `hicache-ratio 4`、`hicache-mem-layout page_first_direct`、`hicache-io-backend direct`
- `hicache-write-policy write_through`、`hicache-storage-backend mooncake`
- **默认 host memory mode(cache),没有 `--hicache-host-memory-mode buffer_only`**
- **没有 `--optimistic-prefill-attempts`**(`#optimistic-req` 为 0)
- L3 SSD:4 TB `mooncake-ssd`,master `--offload_on_evict=false`
D:
- 同并行 tp4/dp4/ep4、`ep-redundant 64`、DSPARK block 3
- `max-running-requests 512`、`cuda-graph-max-bs-decode 512`
- `chunked-prefill-size 32768`、`max-prefill-tokens 8192`
- **没有 `--tokenizer-worker-num`**(默认 1)
- `mem-fraction-static 0.80`
Gateway:
- 实际 live:`--g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256`
- `--g9i-regular-p-queue-reject-secs 0`、`--g9i-decode-max-running 512`
- 网关二进制 hostPath:`/data/ymk/gw-ns-est-td/sgl-model-gateway`
> ⚠️ 平台 `cw-platform.yaml` 的 `Spec.Manifest` 里 gateway 仍是旧的
> `p-queue-reqs-cap 32 / token-cap 16000000 / p-qps 20`;**live RBG 是 64 / 32M / 40**。
> 本次快照与回滚文件都以 live RBG 为准,避免回滚时把网关参数打回旧值。
## 切换前状态
- 所有 Pod `1/1 Running`;P/D/gateway `/health` 200。
- 为切 `buffer_only` 做 L3 SSD 清理前,`/data/ymk/mooncake-ssd` 占用 **2.3T**;清理后 4.0K,`/data` 7.5T available。
- 启动基线(切换后复核):P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`。
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
回滚前确认 `kmctl get cw ds41-gist-exp` 为 `Terminated 0/0`、GPU 0-7 为 0 MiB。
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
File diff suppressed because one or more lines are too long
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T00:16:46Z"
generation: 9
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30735810"
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T12:36:12Z"
message: All roles are ready
observedGeneration: 9
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 9
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T00:16:46Z"
generation: 9
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30735810"
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T12:36:12Z"
message: All roles are ready
observedGeneration: 9
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 9
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,13 @@
# 2026-11-17 cache + gateway no-limit 实验参数(失败记录)
- P: hicache_host_memory_mode=cache, optimistic_prefill_attempts=0, tokenizer-worker-num=16,
hicache ratio 4 / write_through / L3 mooncake, L3 SSD 4T.
- D: tokenizer-worker-num=16.
- Gateway: g9i-td-admit off, p-queue-reqs-cap 0, p-queue-token-cap 0, p-qps 0,
decode-max-running 1000000, disable-circuit-breaker, disable-health-check.
- Image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
- c580 run c580-cache-nolimit-20261117: warmup 4,720/6,385 (74%), 02:33:32 失败
(ServerDisconnected/warmup_failure;P/D/gateway 0 restart,无 503)。
- 观察:P 单批 32k/23k chunk 正常,但每 rank 同时仅 1 个请求;gateway
smg_worker_requests_active 峰值 1,840,P queue 峰值 503,gateway 约 1 核。
- 结论:该 no-limit/no-cb 网关配置不适合作为 c580 基线;仅作异常记录。
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T13:54:16Z"
generation: 6
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "31040141"
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 1000000 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-17T01:58:12Z"
message: All roles are ready
observedGeneration: 6
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 6
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -56,6 +56,7 @@ class AnthropicUsage(BaseModel):
class TextBlock(BaseModel):
type: Literal["text"] = "text"
text: str
cache_control: Optional[dict[str, Any]] = None
class ImageBlock(BaseModel):
@@ -91,6 +92,50 @@ class ToolReferenceBlock(BaseModel):
id: Optional[str] = None
class McpToolReference(BaseModel):
type: Literal["mcp_tool_reference"]
server_name: str
name: str
class McpToolsetReference(BaseModel):
type: Literal["mcp_toolset_reference"]
server_name: str
class ToolDefinition(BaseModel):
"""A tool given by value: `definition` is a `tools` entry."""
type: Literal["tool_definition"]
definition: dict[str, Any]
class ToolAdditionBlock(BaseModel):
"""Anthropic beta block: a ``system`` message offers a tool from its position
on. Rendered through deepseek-recipe, which reads only a ``tool_reference``
target."""
type: Literal["tool_addition"] = "tool_addition"
tool: Annotated[
Union[
ToolReferenceBlock, McpToolReference, McpToolsetReference, ToolDefinition
],
Field(discriminator="type"),
]
cache_control: Optional[dict[str, Any]] = None
class ToolRemovalBlock(BaseModel):
"""Anthropic beta block: a ``system`` message withdraws a tool."""
type: Literal["tool_removal"] = "tool_removal"
tool: Annotated[
Union[ToolReferenceBlock, McpToolReference, McpToolsetReference],
Field(discriminator="type"),
]
cache_control: Optional[dict[str, Any]] = None
class SearchResultBlock(BaseModel):
type: Literal["search_result"] = "search_result"
# ``source`` here is a URL/identifier string (unlike ImageBlock.source).
@@ -117,6 +162,8 @@ AnthropicContentBlock = Annotated[
ToolUseBlock,
ToolResultBlock,
ToolReferenceBlock,
ToolAdditionBlock,
ToolRemovalBlock,
SearchResultBlock,
ThinkingBlock,
RedactedThinkingBlock,
@@ -128,6 +175,8 @@ AnthropicContentBlock = Annotated[
class AnthropicMessage(BaseModel):
role: Literal["user", "assistant", "system"]
content: Union[str, list[AnthropicContentBlock]]
clear_at: Optional[Literal["next_user_message", "never"]] = None
output_config: Optional[dict[str, Any]] = None
# ---------- Tools (discriminated by ``type`` family) ----------
@@ -20,8 +20,10 @@ from sglang.srt.entrypoints.anthropic.protocol import (
AnthropicContentBlock,
AnthropicCountTokensRequest,
AnthropicCountTokensResponse,
AnthropicCustomTool,
AnthropicError,
AnthropicErrorResponse,
AnthropicMessage,
AnthropicMessageEndDelta,
AnthropicMessagesRequest,
AnthropicMessagesResponse,
@@ -32,6 +34,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
ContentBlockStopEvent,
ErrorEvent,
InputJsonDelta,
McpToolReference,
McpToolsetReference,
MessageDeltaEvent,
MessageStartEvent,
MessageStopEvent,
@@ -40,6 +44,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
TextDelta,
ThinkingBlock,
ThinkingDelta,
ToolDefinition,
ToolReferenceBlock,
ToolUseBlock,
is_server_tool,
)
@@ -152,6 +158,207 @@ def _extract_system_text(
return "\n".join(texts) if texts else None
def _block_types(request: AnthropicMessagesRequest) -> set[str]:
"""The block types of the request's system and messages, tool result contents
included."""
types = set()
for content in [request.system, *(message.content for message in request.messages)]:
for block in content if isinstance(content, list) else []:
types.add(block.type)
if block.type == "tool_result" and isinstance(block.content, list):
types.update(item.type for item in block.content)
return types
def _without_cleared_messages(
request: AnthropicMessagesRequest,
) -> AnthropicMessagesRequest:
"""`request` without the turn-scoped system messages (`clear_at:
"next_user_message"`) that a later user message has cleared."""
last_user = max(
(
index
for index, message in enumerate(request.messages)
if message.role == "user"
),
default=-1,
)
rendered = []
for index, message in enumerate(request.messages):
if message.clear_at is not None and message.role != "system":
raise ValueError(
f"messages.{index}.clear_at: clear_at is only permitted on role "
"'system' messages"
)
if message.clear_at == "next_user_message":
_check_turn_scoped(index=index, message=message)
_check_turn_scoped_placement(index=index, messages=request.messages)
if index < last_user:
continue
rendered.append(message)
return request.model_copy(update={"messages": rendered})
def _check_turn_scoped(index: int, message: AnthropicMessage) -> None:
scope = "a turn-scoped system message (clear_at: 'next_user_message')"
if message.output_config is not None:
raise ValueError(f"messages.{index}: output_config is not permitted on {scope}")
for position, block in enumerate(
message.content if isinstance(message.content, list) else []
):
if block.type != "text":
raise ValueError(
f"messages.{index}: a turn-scoped system message supports text "
"blocks only (clear_at: 'next_user_message')"
)
if block.cache_control is not None:
raise ValueError(
f"messages.{index}.content.{position}: cache_control is not "
f"permitted on {scope}"
)
def _check_turn_scoped_placement(index: int, messages: list[AnthropicMessage]) -> None:
# Consecutive system messages are placed as one group.
before = next(
(m.role for m in reversed(messages[:index]) if m.role != "system"), None
)
after = next((m.role for m in messages[index + 1 :] if m.role != "system"), None)
if before != "user" or after == "user":
raise ValueError(
f"messages.{index}: a turn-scoped system message (clear_at: "
"'next_user_message') must follow a user message and either end "
"messages or precede an assistant message"
)
def _declare_inline_tools(
request: AnthropicMessagesRequest,
) -> AnthropicMessagesRequest:
"""`request` with each custom tool a `tool_addition` block defines by value
declared in `tools` as deferred, and the block naming it by reference. A
definition of a tool already offered unchanged is dropped."""
tools = list(request.tools or [])
declared = {tool.name: tool for tool in tools}
offered = {tool.name for tool in tools if not tool.defer_loading}
unresolved: dict[str, str] = {}
messages = []
for index, message in enumerate(request.messages):
if isinstance(message.content, str):
messages.append(message)
continue
blocks = []
for position, block in enumerate(message.content):
where = f"messages.{index}.content.{position}"
for name in _referenced_tool_names(block):
if name not in declared:
unresolved.setdefault(name, where)
if block.type == "tool_addition" and block.tool.type != "tool_reference":
if message.role != "system":
raise ValueError(
f"{where}: tool_addition blocks can only be in system messages"
)
tool = _inline_custom_tool(where=where, target=block.tool)
# Declaring the tool makes it resolvable from the first message on.
if tool.name in unresolved:
raise ValueError(
f"{unresolved[tool.name]}: tool reference {tool.name!r} "
f"precedes the tool_definition that defines it at {where}"
)
earlier = declared.setdefault(tool.name, tool)
if earlier is tool:
tools.append(tool)
elif is_server_tool(earlier) or _definition(earlier) != _definition(
tool
):
raise ValueError(
f"{where}: tool_definition "
f"reuses the name {tool.name!r} with a different "
"definition; replacing a tool mid-conversation is not "
"supported"
)
if tool.name in offered:
continue
offered.add(tool.name)
block = block.model_copy(
update={"tool": ToolReferenceBlock(name=tool.name)}
)
blocks.append(block)
# recipe renders a system message left without blocks as an empty turn.
if blocks or not message.content:
messages.append(message.model_copy(update={"content": blocks}))
return request.model_copy(update={"tools": tools, "messages": messages})
def _referenced_tool_names(block: AnthropicContentBlock) -> list[Optional[str]]:
if block.type == "tool_addition" and block.tool.type == "tool_reference":
return [block.tool.name or block.tool.tool_name]
if block.type == "tool_reference":
return [block.tool_name or block.name]
if block.type == "tool_result" and isinstance(block.content, list):
return [name for item in block.content for name in _referenced_tool_names(item)]
return []
def _inline_custom_tool(
where: str, target: Union[McpToolReference, McpToolsetReference, ToolDefinition]
) -> AnthropicCustomTool:
if target.type != "tool_definition":
raise ValueError(
f"{where}: tool_addition with a {target.type} is not supported; this "
"server runs no MCP connector"
)
kind = target.definition.get("type")
if kind not in (None, "custom"):
raise ValueError(
f"{where}: tool_definition supports custom tools only, got a "
f"definition of type {kind!r}"
)
tool = AnthropicCustomTool.model_validate(target.definition)
if tool.defer_loading:
raise ValueError(
f"{where}: a tool_definition with defer_loading is not supported; "
"declare the tool in tools and add it by reference"
)
return tool.model_copy(update={"defer_loading": True})
def _definition(tool: AnthropicCustomTool) -> dict[str, Any]:
return tool.model_dump(exclude={"type", "defer_loading"})
def _native_block(block: dict[str, Any]) -> dict[str, Any]:
"""`block` with SGLang's accepted aliases spelled as deepseek-recipe reads them."""
kind = block["type"]
if kind == "tool_result":
content = block.get("content")
fields = {
"type": kind,
"tool_use_id": block.get("tool_use_id") or block.get("id"),
"content": (
[_native_block(item) for item in content]
if isinstance(content, list)
else content
),
"is_error": block.get("is_error"),
}
return {key: value for key, value in fields.items() if value is not None}
if kind == "tool_reference":
return {"type": kind, "tool_name": block.get("tool_name") or block.get("name")}
if kind == "tool_addition":
tool = block["tool"]
name = tool.get("name") or tool.get("tool_name")
return {"type": kind, "tool": {"type": "tool_reference", "name": name}}
return block
def _native_message(message: AnthropicMessage) -> dict[str, Any]:
data = message.model_dump(exclude_none=True)
if isinstance(data["content"], list):
data["content"] = [_native_block(block) for block in data["content"]]
return data
def _wrap_sse_event(data: str, event_type: str) -> str:
"""Format an Anthropic SSE event with event type and data lines."""
return f"event: {event_type}\ndata: {data}\n\n"
@@ -218,6 +425,24 @@ class AnthropicServing:
self, anthropic_request: AnthropicMessagesRequest
) -> ChatCompletionRequest:
"""Convert an Anthropic Messages request to an OpenAI ChatCompletion request."""
anthropic_request = _declare_inline_tools(
_without_cleared_messages(anthropic_request)
)
block_types = _block_types(anthropic_request)
if "tool_removal" in block_types:
raise ValueError(
"tool_removal blocks are not supported; a tool cannot be withdrawn "
"mid-conversation"
)
native = self.openai_serving_chat.native_tool_loading and (
bool(block_types & {"tool_reference", "tool_addition"})
or any(
not is_server_tool(tool) and tool.defer_loading
for tool in anthropic_request.tools or []
)
)
if not native and "tool_addition" in block_types:
raise ValueError("tool_addition blocks are not supported by this model")
openai_messages = []
def _convert_anthropic_image_source_to_openai_part(
@@ -719,8 +944,51 @@ class AnthropicServing:
elif chat_request.tools:
chat_request.tool_choice = "auto"
if native:
self._prepare_native_prompt(anthropic_request, chat_request)
return chat_request
def _prepare_native_prompt(
self,
anthropic_request: AnthropicMessagesRequest,
chat_request: ChatCompletionRequest,
) -> None:
"""Render the request's own wire form into `chat_request`, whose tools
become the ones the history has loaded."""
body = {
"messages": [
_native_message(message) for message in anthropic_request.messages
],
# recipe reads a tool with a `type` as a server tool.
"tools": [
tool.model_dump(exclude_none=True, exclude={"type"})
for tool in anthropic_request.tools or []
if not is_server_tool(tool)
],
}
system = anthropic_request.system
if system:
body["system"] = (
system
if isinstance(system, str)
else [block.model_dump(exclude_none=True) for block in system]
)
choice = anthropic_request.tool_choice
if choice is not None and choice.type == "none":
body["tool_choice"] = {"type": "none"}
tools = self.openai_serving_chat.prepare_native_prompt(
chat_request, protocol="messages", body=body
).tools
if (
choice is not None
and choice.type == "tool"
and not any(tool.function.name == choice.name for tool in tools)
):
raise ValueError(
f"tool_choice names {choice.name!r}, a deferred tool no "
"tool_reference or tool_addition block has loaded"
)
async def _handle_non_streaming(
self,
chat_request: ChatCompletionRequest,
@@ -3,7 +3,7 @@ concrete adapters load only when selected."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol, Union
import msgspec
@@ -15,8 +15,27 @@ if TYPE_CHECKING:
ToolCall,
)
from sglang.srt.function_call.core_types import ToolCallItem
from sglang.srt.multimodal.media_io import ImageData
EncodingBackend = Literal["builtin", "recipe"]
HistoryToolCallArguments = Literal["strict", "lenient", "as_sent"]
NativeProtocol = Literal["responses", "messages"]
class NativePrompt(msgspec.Struct, frozen=True, kw_only=True):
"""A Responses or Messages request rendered from its own wire form."""
prompt_ids: list[int]
tools: list[Tool]
"""The tools the model can call after the request's history: those loaded from
the start and those the history has loaded since."""
image_data: list[ImageData] = []
"""The images the prompt's placeholders stand for, in order."""
namespace_names: dict[str, tuple[str, str]] = {}
"""The call name of each namespace member, mapped to its namespace and the
name the request gives it there."""
continued_turn: Optional[dict[str, Any]] = None
"""The final assistant message the prompt continues, as a chat message."""
class ChatOutputHooks(msgspec.Struct, frozen=True, kw_only=True):
@@ -78,6 +97,25 @@ class ChatEncodingOutputAdapter(Protocol):
) -> ResponsesOutputStream: ...
class NativeRequestEncoder(Protocol):
def encode_native(
self,
*,
protocol: NativeProtocol,
body: dict[str, Any],
thinking: bool,
effort: Union[str, int, float, None],
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> NativePrompt:
"""``body`` carries the request's conversation, tools and, for
``tool_choice="none"``, its choice; ``thinking`` and ``effort`` are the
handler's decisions, as for ``encode_messages``. ``response_schema`` is
the JSON schema the answer follows; a trailing assistant turn is continued
under ``continue_final_message`` and otherwise re-sent as a user turn."""
...
class ChatEncodingAdapter(Protocol):
collects_openai_media: bool
"""Whether media still has to be collected from the OpenAI messages after
@@ -92,6 +130,12 @@ class ChatEncodingAdapter(Protocol):
"""None when the configured parsers leave output to the text parsers."""
...
@property
def native_encoder(self) -> Optional[NativeRequestEncoder]:
"""None when Responses and Messages requests that load tools mid-conversation
have no rendering of their own wire form."""
...
def encode_messages(
self,
*,
@@ -17,6 +17,7 @@ from __future__ import annotations
import logging
import time
import msgspec
import unicodedata
import uuid
from dataclasses import dataclass
@@ -743,6 +744,10 @@ class ChatCompletionMessageGenericParam(BaseModel):
name: Optional[str] = None
phase: Optional[Literal["commentary", "final_answer"]] = None
reasoning_content: Optional[str] = None
action: Optional[str] = Field(
default=None,
description="Action tag for assistant messages (used by deepseek-recipe).",
)
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
# DeepSeek API prefix completion: continue this final assistant message.
@@ -796,14 +801,22 @@ class Function(BaseModel):
description: Optional[str] = Field(default=None, examples=[None])
name: str
parameters: Optional[object] = None
strict: bool = False
strict: Optional[bool] = None
defer_loading: Optional[bool] = None
namespace: Optional[str] = Field(
default=None,
description="Tool namespace for disambiguation (used by deepseek-recipe).",
)
@model_serializer(mode="wrap")
def _serialize(self, handler):
data = handler(self)
if self.defer_loading is None:
data.pop("defer_loading", None)
if self.strict is None:
data.pop("strict", None)
if self.namespace is None:
data.pop("namespace", None)
return data
@@ -821,6 +834,26 @@ class Tool(BaseModel):
return self
class ChatCompletionNamespaceTool(BaseModel):
"""A Responses-style namespace grouping sent to the Chat Completions API.
Chat Completions has no namespace field in its tool-call shape, so the
members are flattened to ``<namespace>__<function>`` before rendering; the
model calls that flat name and the response reports it back unchanged.
"""
type: Literal["namespace"]
name: str
description: Optional[str] = None
tools: List[Tool]
@model_validator(mode="after")
def validate_members(self) -> "ChatCompletionNamespaceTool":
if not self.tools:
raise ValueError("namespace tools must list at least one function tool")
return self
# Tool is defined after the message params that reference it, so the forward
# reference has to be resolved explicitly.
ChatCompletionMessageGenericParam.model_rebuild()
@@ -908,11 +941,22 @@ class ChatCompletionRequest(PDRoutingFields):
temperature: Optional[float] = None
top_p: Optional[float] = None
user: Optional[str] = None
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
tools: Optional[List[Union[Tool, ChatCompletionNamespaceTool]]] = Field(
default=None, examples=[None]
)
# Set by /v1/responses when the request has namespace tools: the chat template
# renders these, while ``tools`` (flattened to ``<namespace>.<function>``) still
# drives tool-call parsing.
_template_tools: Optional[List[Dict[str, Any]]] = PrivateAttr(default=None)
# Set with ``input_ids`` by a caller that rendered them itself: the images the
# prompt's placeholders stand for, in order.
_input_image_data: Optional[List[Any]] = PrivateAttr(default=None)
# Whether ``tools`` are the only tools the rendered prompt shows, so a call
# to any other tool is dropped rather than returned.
_drops_unlisted_tool_calls: bool = PrivateAttr(default=False)
# Set with ``input_ids`` under ``continue_final_message``: the final assistant
# message the prompt continues, where output parsing resumes.
_continued_turn: Optional[Dict[str, Any]] = PrivateAttr(default=None)
tool_choice: Union[ToolChoice, Literal["auto", "required", "none"]] = Field(
default="auto", examples=["none"]
) # noqa
@@ -1140,6 +1184,39 @@ class ChatCompletionRequest(PDRoutingFields):
return values
@model_validator(mode="after")
def flatten_namespace_tools(self) -> "ChatCompletionRequest":
"""Render namespace groups as flat ``ns__function`` chat tools."""
if not self.tools:
return self
flattened: List[Tool] = []
for tool in self.tools:
if isinstance(tool, ChatCompletionNamespaceTool):
for member in tool.tools:
name = f"{tool.name}__{member.function.name}"
description = member.function.description
if tool.description:
description = (
tool.description + chr(10) + description
if description
else tool.description
)
flattened.append(
Tool(
type="function",
function=Function(
name=name,
description=description,
parameters=member.function.parameters,
strict=member.function.strict,
),
)
)
else:
flattened.append(tool)
self.tools = flattened
return self
def to_sampling_params(
self,
stop: List[str],
@@ -1252,6 +1329,7 @@ class ChatMessage(BaseModel):
role: Optional[str] = None
content: Optional[str] = None
reasoning_content: Optional[str] = None
action: Optional[str] = None
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
@model_serializer(mode="wrap")
@@ -1259,6 +1337,8 @@ class ChatMessage(BaseModel):
data = handler(self)
if self.tool_calls is None:
data.pop("tool_calls", None)
if self.action is None:
data.pop("action", None)
return data
@@ -1811,6 +1891,8 @@ RESPONSE_TOOL_TYPES = Literal[
class ResponseTool(BaseModel):
"""Tool definition for responses."""
model_config = ConfigDict(extra="allow")
type: RESPONSE_TOOL_TYPES = Field(description="Type of tool to enable")
name: Optional[str] = None
description: Optional[str] = None
@@ -1830,6 +1912,40 @@ class ResponseTool(BaseModel):
return self
class ResponseToolState(msgspec.Struct, frozen=True):
"""Public tool metadata and name mappings for one translated request."""
public_tools: list[ResponseTool] | None = None
public_choice: str | dict[str, Any] | None = None
names: dict[str, dict[str, str]] = {}
# The tools the model can call after the history, from a request rendered in
# its own wire form; None when `tools` alone declares them.
chat_tools: list[Tool] | None = None
# ``custom`` tools a ``tool_search_output`` loaded, beside the declared ones.
discovered_custom_names: frozenset[str] = frozenset()
class ResponseToolSearchCall(BaseModel):
"""A client-executed ``tool_search`` call; the pinned openai SDK has no type."""
id: str
type: Literal["tool_search_call"] = "tool_search_call"
call_id: str
execution: Literal["client"] = "client"
arguments: Dict[str, Any]
status: Literal["in_progress", "completed", "incomplete"]
class ResponseToolSearchCallEvent(BaseModel):
"""``response.output_item.added`` / ``.done`` for a tool search call, which the
SDK's event types reject."""
type: Literal["response.output_item.added", "response.output_item.done"]
sequence_number: int
output_index: int
item: ResponseToolSearchCall
class ResponseInputMessageParam(EasyInputMessageParam, total=False):
phase: Optional[Literal["commentary", "final_answer"]]
@@ -1856,6 +1972,10 @@ ResponseInputOutputItem: TypeAlias = Union[
class ResponsesRequest(PDRoutingFields):
"""Request body for v1/responses endpoint."""
_tool_state: ResponseToolState = PrivateAttr(default_factory=ResponseToolState)
# By call_id, the name the model called a tool by when the output item
# reports another; the stored history replays the call under it.
_called_tool_names: Dict[str, str] = PrivateAttr(default_factory=dict)
_response_parser_prefix: str = PrivateAttr(default="")
_response_parser_grammar_start: Optional[str] = PrivateAttr(default=None)
_response_parser_json_response_format: bool = PrivateAttr(default=False)
@@ -2052,7 +2172,10 @@ class ResponsesRequest(PDRoutingFields):
"name"
)
if tool_choice.get("type") in ("function", "custom") and name:
return {"type": "function", "name": name}
choice = {"type": tool_choice["type"], "name": name}
if "namespace" in tool_choice:
choice["namespace"] = tool_choice["namespace"]
return choice
return "auto"
def to_sampling_params(
@@ -2176,6 +2299,7 @@ class ResponsesResponse(BaseModel):
ResponseOutputItem,
ResponseReasoningItem,
ResponseFunctionToolCall,
ResponseToolSearchCall,
]
] = Field(default_factory=list)
status: Literal[
@@ -2241,6 +2365,7 @@ class ResponsesResponse(BaseModel):
ResponseOutputItem,
ResponseReasoningItem,
ResponseFunctionToolCall,
ResponseToolSearchCall,
]
],
status: str,
@@ -957,12 +957,13 @@ class OpenAIServingChat(OpenAIServingBase):
The reasoning parser is told the same thing, so channel-framed formats
keep their framing intact exactly when a tool-call parser consumes it.
"""
return bool(
request.tool_choice != "none"
and self._effective_tools(request)
and self.tool_call_parser
and not request._response_parser_json_response_format
)
if not self.tool_call_parser or request._response_parser_json_response_format:
return False
if request._drops_unlisted_tool_calls and request.tool_choice == "none":
# The history showed the model tools, so it can still write a call;
# parsing drops it, where skipping the parser would return its markup.
return True
return bool(request.tool_choice != "none" and self._effective_tools(request))
def _validate_request(self, request: ChatCompletionRequest) -> str | None:
"""Validate that the input is valid."""
@@ -1367,18 +1368,7 @@ class OpenAIServingChat(OpenAIServingBase):
self, request: ChatCompletionRequest, is_multimodal: bool
) -> MessageProcessingResult:
"""Process chat messages and apply chat template"""
if self.default_chat_template_kwargs:
ctk = dict(request.chat_template_kwargs or {})
for k, v in self.default_chat_template_kwargs.items():
ctk.setdefault(k, v)
request.chat_template_kwargs = ctk
effort = ctk.get("reasoning_effort")
if effort is not None and request.reasoning_effort is None:
request.reasoning_effort = effort
normalize_hunyuan_reasoning_effort(
request, self.reasoning_parser, self.template_manager.reasoning_config
)
self._apply_request_defaults(request)
# GptOss model needs to keep special tokens for harmony parsing
if self.is_gpt_oss or self.is_gemma4:
@@ -1484,7 +1474,7 @@ class OpenAIServingChat(OpenAIServingBase):
result = MessageProcessingResult(
prompt="",
prompt_ids=request.input_ids,
image_data=None,
image_data=request._input_image_data,
audio_data=None,
video_data=None,
modalities=[],
@@ -1540,6 +1530,64 @@ class OpenAIServingChat(OpenAIServingBase):
result.reasoning_end_token_ids = list(token_ids)
return result
def _apply_request_defaults(self, request: ChatCompletionRequest) -> None:
"""Settle the thinking and effort that rendering reads; repeating it
changes nothing."""
if self.default_chat_template_kwargs:
ctk = dict(request.chat_template_kwargs or {})
for k, v in self.default_chat_template_kwargs.items():
ctk.setdefault(k, v)
request.chat_template_kwargs = ctk
effort = ctk.get("reasoning_effort")
if effort is not None and request.reasoning_effort is None:
request.reasoning_effort = effort
normalize_hunyuan_reasoning_effort(
request, self.reasoning_parser, self.template_manager.reasoning_config
)
@property
def native_tool_loading(self) -> bool:
"""Whether Responses and Messages requests that load tools mid-conversation
render from their own wire form (see `prepare_native_prompt`)."""
return (
self._encoding_adapter is not None
and self._encoding_adapter.native_encoder is not None
)
def prepare_native_prompt(
self,
request: ChatCompletionRequest,
*,
protocol: chat_encoding_adapters.NativeProtocol,
body: dict[str, Any],
) -> chat_encoding_adapters.NativePrompt:
"""Render `body`, the Responses or Messages form of `request`, into its
`input_ids` and images, and set its `tools` to the tools the model can call
after the history; rendering then keeps only grammar, stops and parsing."""
self._apply_request_defaults(request)
response_format = request.response_format
prompt = self._encoding_adapter.native_encoder.encode_native(
protocol=protocol,
body=body,
thinking=self._get_reasoning_from_request(request),
effort=request.reasoning_effort,
response_schema=(
response_format.json_schema.schema_
if response_format is not None and response_format.json_schema
else None
),
continue_final_message=request.continue_final_message,
)
if prompt.image_data and not self.tokenizer_manager.model_config.is_multimodal:
raise ValueError("image input is not supported for this model")
request.input_ids = prompt.prompt_ids
request.tools = prompt.tools or None
request._input_image_data = prompt.image_data or None
request._drops_unlisted_tool_calls = True
request._continued_turn = prompt.continued_turn
return prompt
def _apply_jinja_template(
self,
request: ChatCompletionRequest,
@@ -5,15 +5,20 @@ request semantics."""
from __future__ import annotations
from collections.abc import Sequence
import re
from typing import TYPE_CHECKING, Any, Optional, Union
from sglang.srt.entrypoints.openai.chat_encoding_adapters import (
ADAPTER_OUTPUT_PARSERS,
NativePrompt,
NativeProtocol,
select_new_output_ids,
token_output_sse_chunks,
)
from sglang.srt.entrypoints.openai.protocol import (
Function,
FunctionResponse,
Tool,
ToolCall,
)
from sglang.srt.environ import envs
@@ -26,7 +31,9 @@ from sglang.srt.parser.deepseek_recipe_output import (
whole_tool_calls,
)
from sglang.srt.parser.deepseek_recipe_renderer import (
BUDGET_TIER,
DeepSeekRecipe,
loaded_tools,
parse_effort_budget,
)
from sglang.srt.runtime_context import get_serving
@@ -37,7 +44,6 @@ if TYPE_CHECKING:
from sglang.srt.entrypoints.openai.protocol import (
ChatCompletionRequest,
ResponsesRequest,
Tool,
)
DEFAULT_EFFORT_ENVS = {
@@ -45,6 +51,12 @@ DEFAULT_EFFORT_ENVS = {
"dsv41": envs.SGLANG_DSV41_REASONING_EFFORT,
}
JSON_RESPONSE_FORMATS = ("json_object", "json_schema")
# V4 renders deferred tools up front and never loads one mid-conversation.
NATIVE_TOOL_LOADING_SPEC = "dsv41"
NATIVE_EFFORT_FIELDS: dict[str, str] = {
"responses": "reasoning",
"messages": "output_config",
}
class DeepSeekRecipeChatEncodingAdapter:
@@ -96,6 +108,13 @@ class DeepSeekRecipeChatEncodingAdapter:
def output(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
return self if self._token_output else None
def validate_request(self, request: ChatCompletionRequest) -> Optional[str]:
return None
@property
def native_encoder(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
return self if self._spec == NATIVE_TOOL_LOADING_SPEC else None
def encode_messages(
self,
*,
@@ -149,6 +168,66 @@ class DeepSeekRecipeChatEncodingAdapter:
prompt_ids += self.deepseek_recipe.encode_text(task_prefix)
return prompt_ids
def encode_native(
self,
*,
protocol: NativeProtocol,
body: dict[str, Any],
thinking: bool,
effort: Union[str, int, float, None],
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> NativePrompt:
tier, budget = self._effort(effort)
# recipe reads an effort as a request to think.
if thinking and tier is not None:
body = {**body, NATIVE_EFFORT_FIELDS[protocol]: {"effort": tier}}
conversation = self.deepseek_recipe.with_host_settings(
self.deepseek_recipe.convert_native(protocol, body, thinking=thinking),
response_schema=response_schema,
continue_final_message=continue_final_message,
)
last = conversation.messages[-1] if conversation.messages else None
# recipe's Messages conversion drops `strict`, which SGLang's grammar enforces.
strict = {tool.get("name") for tool in body["tools"] if tool.get("strict")}
tools = [
Tool(
type="function",
function=Function(
name=tool.name,
description=tool.description,
parameters=tool.parameters,
strict=True if tool.name in strict else tool.strict,
),
)
for tool in loaded_tools(conversation)
]
return NativePrompt(
prompt_ids=self.deepseek_recipe.encode_conversation(
conversation, effort_budget=budget
),
tools=tools,
image_data=[
ImageData(url=url)
for url in self.deepseek_recipe.image_urls(conversation)
],
namespace_names=(
self.deepseek_recipe.namespace_tool_names(_responses_namespaces(body))
if protocol == "responses"
else {}
),
continued_turn=(
{"role": "assistant", "content": last.content}
| (
{"reasoning_content": last.reasoning_content}
if last.reasoning_content is not None
else {}
)
if last is not None and last.prefix
else None
),
)
def _encode(
self,
body: dict[str, Any],
@@ -157,21 +236,29 @@ class DeepSeekRecipeChatEncodingAdapter:
task: Optional[str] = None,
image_urls: Optional[list[str]] = None,
) -> list[int]:
if effort is None or effort == "none":
effort = self._default_effort
budget = None
if effort is not None and not isinstance(effort, str):
if self._spec != "dsv41":
raise ValueError(
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
)
budget = parse_effort_budget(effort)
elif effort is not None:
body["reasoning_effort"] = effort
tier, budget = self._effort(effort)
if budget is None and tier is not None:
body["reasoning_effort"] = tier
return self.deepseek_recipe.encode(
body, task=task, effort_budget=budget, image_urls=image_urls
)
def _effort(
self, effort: Union[str, int, float, None]
) -> tuple[Optional[str], Optional[int]]:
"""The named tier recipe renders and, for a V4.1 integer budget, the budget
that replaces the tier's line."""
effort = _numeric_effort(effort)
if effort is None or effort == "none":
effort = self._default_effort
if effort is None or isinstance(effort, str):
return effort, None
if self._spec != "dsv41":
raise ValueError(
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
)
return BUDGET_TIER, parse_effort_budget(effort)
def parse_chat_response(
self,
*,
@@ -320,7 +407,7 @@ class DeepSeekRecipeChatEncodingAdapter:
self, request: ChatCompletionRequest
) -> DeepSeekRecipeOutputParser:
response_format = request.response_format
continued = (
continued = request._continued_turn or (
_trailing_assistant_turn(
[
message.model_dump(exclude_unset=True, by_alias=True, mode="json")
@@ -344,6 +431,11 @@ class DeepSeekRecipeChatEncodingAdapter:
separate_reasoning=self._separate_reasoning and request.separate_reasoning,
stream_reasoning=request.stream_reasoning,
stop=_stop_strings(request.stop),
callable_names=(
_native_callable_names(request.tools or [], request.tool_choice)
if request._drops_unlisted_tool_calls
else None
),
)
def _new_responses_parser(
@@ -353,12 +445,25 @@ class DeepSeekRecipeChatEncodingAdapter:
separate_reasoning: bool,
parse_tool_calls: bool,
) -> DeepSeekRecipeOutputParser:
native = request._tool_state.chat_tools is not None
return DeepSeekRecipeOutputParser(
self.deepseek_recipe,
thinking_stage=self.deepseek_recipe.parse_start(thinking=require_reasoning),
parse_tool_calls=parse_tool_calls,
# Under `none` the tool parser runs only to drop calls; a JSON answer
# must be read as JSON, or a call marker spelled in a string ends it.
parse_json_output=native
and request.tool_choice == "none"
and request.has_json_schema_constraint(),
separate_reasoning=separate_reasoning,
stop=_stop_strings(request.stop),
callable_names=(
_native_callable_names(
request._tool_state.chat_tools, request.tool_choice
)
if native
else None
),
)
def _tool_call_id(self, index: int, name: str, history_tool_calls_cnt: int) -> str:
@@ -390,6 +495,37 @@ class DeepSeekRecipeResponsesStream:
return delta.reasoning, delta.content, _tool_call_items(delta.tool_calls)
def _responses_namespaces(body: dict[str, Any]) -> list[dict[str, Any]]:
"""The namespace tools a Responses body declares and its history discovers, in
the order recipe's conversion names their members."""
discovered = [
tool
for item in body["input"]
if isinstance(item, dict) and item.get("type") == "tool_search_output"
for tool in item["tools"]
]
return [
tool for tool in [*body["tools"], *discovered] if tool["type"] == "namespace"
]
def _null_content_as_empty(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""As `_process_messages` renders them: a null content is ""."""
return [
message if message.get("content") is not None else {**message, "content": ""}
for message in messages
]
def _numeric_effort(
effort: Union[str, int, float, None],
) -> Union[str, int, float, None]:
"""A numeric string effort ("30", "0.3") as the number it spells."""
if isinstance(effort, str) and re.fullmatch(r"\d+(\.\d+)?", effort.strip()):
return float(effort) if "." in effort else int(effort)
return effort
def _trailing_assistant_turn(
messages: list[dict[str, Any]],
) -> Optional[dict[str, Any]]:
@@ -405,6 +541,18 @@ def _stop_strings(stop: Union[str, list[str], None]) -> list[str]:
return [stop] if isinstance(stop, str) else stop or []
def _native_callable_names(
tools: list[Tool], tool_choice: Any
) -> Optional[frozenset[str]]:
"""The names of `tools`, all that a natively rendered prompt shows the model;
None when calls to other tools are returned. `none` allows no call at all."""
if tool_choice == "none":
return frozenset()
if envs.SGLANG_FORWARD_UNKNOWN_TOOLS.get():
return None
return frozenset(tool.function.name for tool in tools)
def _tool_call_items(
calls: Sequence[Union[RecipeToolCall, RecipeToolCallDelta]],
) -> list[ToolCallItem]:
@@ -36,7 +36,10 @@ if TYPE_CHECKING:
class InklingChatEncodingAdapter:
collects_openai_media = True
strict_tool_call_arguments = True
history_tool_call_arguments = "strict"
renders_response_format = False
uncounted_prompt_tokens = 0
native_encoder = None
def __init__(
self,
@@ -13,6 +13,7 @@ from http import HTTPStatus
from typing import TYPE_CHECKING, Any, AsyncGenerator, AsyncIterator, Optional, Union
import jinja2
import msgspec
import openai.types.responses as openai_responses_types
import orjson
from fastapi import Request
@@ -43,6 +44,7 @@ from openai.types.responses.response_reasoning_summary_part_done_event import (
Part as ResponseReasoningSummaryDonePart,
)
from openai_harmony import Message as OpenAIMessage
from pydantic import BaseModel, TypeAdapter
from sglang.srt.entrypoints.context import (
ConversationContext,
@@ -70,7 +72,12 @@ from sglang.srt.entrypoints.openai.protocol import (
ResponseOutputMessage,
ResponsesRequest,
ResponsesResponse,
ResponseTool,
ResponseToolSearchCall,
ResponseToolSearchCallEvent,
Tool,
ToolChoice,
ToolChoiceFuncName,
UsageInfo,
)
from sglang.srt.entrypoints.openai.responses_adapters import (
@@ -114,6 +121,100 @@ class _MediaInputValidationError(ValueError):
pass
class ResponseToolChoiceError(ValueError):
"""A response tool_choice that names a tool the model cannot call."""
TOOL_SEARCH = "tool_search"
TOOL_SEARCH_ITEMS = ("tool_search_call", "tool_search_output")
APPLY_PATCH = "apply_patch"
NAMESPACE_MEMBERS_ERROR = (
"namespace tools must list function tools, each with a type and a name"
)
def _holds_function_tools(members: Any) -> bool:
return isinstance(members, list) and all(
isinstance(member, dict)
and member.get("type") == "function"
and isinstance(member.get("name"), str)
for member in members
)
def _defers_loading(tool: ResponseTool) -> bool:
if tool.type == "namespace":
return bool(tool.defer_loading) or any(
isinstance(member, dict) and member.get("defer_loading")
for member in tool.tools or []
)
return tool.type in ("function", "custom") and bool(tool.defer_loading)
def _defer_tool_loading(
tools: list[dict[str, Any]], items: list
) -> tuple[list[dict[str, Any]], list]:
"""`tools` without those that defer loading, and `items` whose
`tool_search_output` entries naming a deferred tool take the fields they
leave out from its declaration, so a search result loads it where it returns.
recipe's Responses conversion renders a deferred tool up front and rejects a
search result that repeats a declared name."""
deferred: dict[tuple[Optional[str], str], dict[str, Any]] = {}
shown = []
for tool in tools:
if tool["type"] == "namespace":
members = []
for member in tool["tools"]:
if tool.get("defer_loading") or member.get("defer_loading"):
deferred[(tool["name"], member["name"])] = member
else:
members.append(member)
if members:
shown.append({**tool, "tools": members})
elif tool["type"] in ("function", "custom") and tool.get("defer_loading"):
deferred[(None, tool["name"])] = tool
else:
shown.append(tool)
def loaded(tool: dict[str, Any], namespace: Optional[str] = None) -> dict:
if tool["type"] == "namespace":
members = [loaded(member, tool["name"]) for member in tool["tools"]]
return {**tool, "tools": members}
declared = deferred.get((namespace, tool.get("name")))
if declared is None or declared["type"] != tool["type"]:
return tool
return declared | {k: v for k, v in tool.items() if v is not None}
if deferred:
items = [
(
{**item, "tools": [loaded(tool) for tool in item["tools"]]}
if isinstance(item, dict) and item.get("type") == "tool_search_output"
else item
)
for item in items
]
return shown, items
def _tool_search_call(
item_id: str, call_id: str, arguments: str
) -> ResponseToolSearchCall:
"""A search call reports object arguments; one whose arguments are not a JSON
object is incomplete."""
try:
parsed = json.loads(arguments)
except ValueError:
parsed = None
complete = isinstance(parsed, dict)
return ResponseToolSearchCall(
id=item_id,
call_id=call_id,
arguments=parsed if complete else {},
status="completed" if complete else "incomplete",
)
def _build_output_text_logprobs(meta_info: dict) -> list[Logprob]:
"""Reshape decoded ``meta_info`` logprobs into the Responses logprob type,
covering every generated token."""
@@ -319,9 +420,21 @@ class OpenAIServingResponses(OpenAIServingChat):
# FIXME: If the engine is dead, raise an error
# This is required for the streaming case
# Such a request's callable tools are known once its history renders,
# so `_make_request` checks its tool_choice instead.
native = self._renders_natively(request)
if native:
tools_error = self._native_tools_error(request)
if tools_error is not None:
return self.create_error_response(tools_error, param="tools")
# ``tool_choice="required"`` needs a tool the parser can actually force.
if request.tool_choice == "required" and not any(
tool.type in ("function", "custom") for tool in (request.tools or [])
if (
not native
and request.tool_choice == "required"
and not any(
tool.type in ("function", "custom") for tool in (request.tools or [])
)
):
return self.create_error_response(
'tool_choice="required" requires at least one tool with '
@@ -330,9 +443,13 @@ class OpenAIServingResponses(OpenAIServingChat):
)
tool_choice = request.effective_tool_choice()
if isinstance(tool_choice, dict) and not any(
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
for tool in request.tools or []
if (
not native
and isinstance(tool_choice, dict)
and not any(
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
for tool in request.tools or []
)
):
return self.create_error_response(
f"Tool {tool_choice['name']!r} is not declared in tools",
@@ -415,6 +532,8 @@ class OpenAIServingResponses(OpenAIServingChat):
except _MediaInputValidationError as e:
return self.create_error_response(str(e))
except ResponseToolChoiceError as e:
return self.create_error_response(str(e), param="tool_choice")
except (ValueError, TypeError, RuntimeError, jinja2.TemplateError) as e:
logger.exception("Error in preprocessing prompt inputs")
return self.create_error_response(f"{e} {e.__cause__}")
@@ -684,7 +803,10 @@ class OpenAIServingResponses(OpenAIServingChat):
prev_response: Optional[ResponsesResponse],
tokenizer: Any,
):
messages = self._construct_input_messages(request, prev_response)
native = self._renders_natively(request)
messages = (
[] if native else self._construct_input_messages(request, prev_response)
)
chat_tools = self._response_tools_to_chat_tools(request)
chat_request = ChatCompletionRequest(
@@ -707,6 +829,8 @@ class OpenAIServingResponses(OpenAIServingChat):
chat_template_kwargs=request.chat_template_kwargs,
)
chat_request._template_tools = self._response_tools_to_template_tools(request)
if native:
self._prepare_native_responses_prompt(request, chat_request)
media_error = self._validate_media_content(chat_request)
if media_error:
@@ -906,10 +1030,7 @@ class OpenAIServingResponses(OpenAIServingChat):
else:
self.msg_store[response.id] = self._response_input_history(
request
) + [
item.model_dump(exclude_none=True)
for item in response.output
]
) + self._replayed_output_items(request, response.output)
return response
@@ -952,12 +1073,18 @@ class OpenAIServingResponses(OpenAIServingChat):
name: str,
arguments: str,
custom_names: set[str],
namespace_names: Optional[dict[str, tuple[str, str]]] = None,
) -> Union[ResponseFunctionToolCall, ResponseCustomToolCall]:
tool_names: dict[str, dict[str, str]],
*,
client_tool_search: bool = False,
) -> Union[
ResponseFunctionToolCall, ResponseCustomToolCall, ResponseToolSearchCall
]:
"""A call against a ``custom`` tool reports its freeform payload rather
than the JSON arguments of the shim function tool. A call against a
namespace member reports ``namespace`` and the bare function ``name``."""
call_id = f"call_{random_uuid()[:24]}"
if client_tool_search and name == TOOL_SEARCH:
return _tool_search_call(f"tsc_{random_uuid()[:8]}", call_id, arguments)
if name in custom_names:
return ResponseCustomToolCall(
type="custom_tool_call",
@@ -966,7 +1093,13 @@ class OpenAIServingResponses(OpenAIServingChat):
name=name,
input=decode_custom_tool_input(arguments),
)
namespace, name = (namespace_names or {}).get(name, (None, name))
entry = (tool_names or {}).get(name)
if isinstance(entry, tuple):
namespace, name = entry
elif isinstance(entry, dict):
namespace, name = entry.get("namespace"), entry.get("name", name)
else:
namespace = None
return ResponseFunctionToolCall(
arguments=arguments,
call_id=call_id,
@@ -1063,12 +1196,17 @@ class OpenAIServingResponses(OpenAIServingChat):
finish_reason: Optional[dict[str, Any]] = None,
):
chat_tools = self._response_tools_to_chat_tools(request)
custom_names = custom_tool_names(request.tools)
custom_names = self._custom_tool_names(request)
namespace_names = (
namespace_function_names(request.tools)
if self.tool_call_parser == "muse"
else {}
else {
call_name: (entry.get("namespace"), entry.get("name", call_name))
for call_name, entry in request._tool_state.names.items()
if entry.get("namespace")
}
)
client_tool_search = self._client_tool_search(request)
encoding_output = self._encoding_output()
if encoding_output is not None:
required_json = self._uses_required_tool_json(request, chat_tools)
@@ -1088,12 +1226,17 @@ class OpenAIServingResponses(OpenAIServingChat):
content, custom_names, namespace_names
)
else:
tool_call_items = [
self._make_tool_call_item(
call.name, call.parameters, custom_names, namespace_names
tool_call_items = []
for call in calls:
item = self._make_tool_call_item(
call.name,
call.parameters,
custom_names,
namespace_names,
client_tool_search=client_tool_search,
)
for call in calls
]
self._record_called_name(request, item.call_id, call.name)
tool_call_items.append(item)
else:
reasoning_content, content, tool_call_items = self._parse_text_output(
request,
@@ -1194,14 +1337,15 @@ class OpenAIServingResponses(OpenAIServingChat):
try:
content, call_info_list = parser.parse_non_stream(content)
for call_info in call_info_list:
tool_call_items.append(
self._make_tool_call_item(
call_info.name,
call_info.parameters or "",
custom_names,
namespace_names,
)
item = self._make_tool_call_item(
call_info.name,
call_info.parameters or "",
custom_names,
namespace_names,
client_tool_search=self._client_tool_search(request),
)
self._record_called_name(request, item.call_id, call_info.name)
tool_call_items.append(item)
parsed_via_native = bool(call_info_list)
except Exception as e:
logger.error("Tool call parsing error: %s", e)
@@ -1268,6 +1412,10 @@ class OpenAIServingResponses(OpenAIServingChat):
def _tool_call_parsing_enabled(
self, request: ResponsesRequest, chat_tools: list[Tool]
) -> bool:
if request._tool_state.chat_tools is not None and request.tool_choice == "none":
# The history showed the model tools, so it can still write a call;
# parsing drops it, where skipping the parser would return its markup.
return bool(self.tool_call_parser)
return bool(
chat_tools and self.tool_call_parser and request.tool_choice != "none"
)
@@ -1326,7 +1474,213 @@ class OpenAIServingResponses(OpenAIServingChat):
return tool_choice
return {"type": "function", "function": {"name": tool_choice["name"]}}
def _renders_natively(self, request: ResponsesRequest) -> bool:
"""Whether the served encoding renders the request in its own wire form,
which tool search, namespace tools and deferred tools take."""
return self.native_tool_loading and (
any(
tool.type == "namespace" or _defers_loading(tool)
for tool in request.tools
)
or self._uses_tool_search(request)
)
def _uses_tool_search(self, request: ResponsesRequest) -> bool:
"""Whether the request searches for tools, or its history has."""
if any(tool.type == TOOL_SEARCH for tool in request.tools):
return True
return any(
isinstance(item, dict) and item.get("type") in TOOL_SEARCH_ITEMS
for item in self._history_items(request)
)
def _history_items(self, request: ResponsesRequest) -> list:
"""The input history, before `previous_response_id` is known to exist."""
stored = self.msg_store.get(request.previous_response_id) or []
return list(stored) + (request.input if isinstance(request.input, list) else [])
def _native_tools_error(self, request: ResponsesRequest) -> Optional[str]:
deferred_types = {
tool.name: tool.type
for tool in request.tools
if tool.type in ("function", "custom") and tool.defer_loading
}
for tool in request.tools:
if (
tool.type == TOOL_SEARCH
and (tool.model_extra or {}).get("execution") != "client"
):
return (
'tool_search requires execution "client": this server runs no '
"searches"
)
if tool.type == "namespace" and not _holds_function_tools(tool.tools):
return NAMESPACE_MEMBERS_ERROR
for item in self._history_items(request):
if not isinstance(item, dict) or item.get("type") != "tool_search_output":
continue
tools = item.get("tools")
if not isinstance(tools, list) or not all(
isinstance(tool, dict)
and tool.get("type") in ("function", "namespace", "custom")
for tool in tools
):
return (
"tool_search_output tools must be a list of function, namespace "
"or custom tools"
)
for tool in tools:
if tool["type"] == "namespace" and not _holds_function_tools(
tool.get("tools")
):
return NAMESPACE_MEMBERS_ERROR
# The served encoding renders no other custom tool.
if tool["type"] == "custom" and tool.get("name") != APPLY_PATCH:
return (
f"tool_search_output can load only the {APPLY_PATCH!r} "
"custom tool"
)
# A deferred declaration is loaded as the tool it declares.
declared = deferred_types.get(tool.get("name"), tool["type"])
if tool["type"] != "namespace" and declared != tool["type"]:
return (
f"tool_search_output returns {tool.get('name')!r} as a "
f"{tool['type']} tool, but tools declares it as a deferred "
f"{declared} tool"
)
return None
def _prepare_native_responses_prompt(
self, request: ResponsesRequest, chat_request: ChatCompletionRequest
) -> None:
"""Render the request's own wire form into `chat_request`, whose tools and
choice become the ones the history has loaded."""
choice = request.effective_tool_choice()
tools, items = _defer_tool_loading(
[tool.model_dump(exclude_unset=True) for tool in request.tools],
[
item
for item in map(
self._native_input_item, self._response_input_history(request)
)
if item is not None
],
)
body = {"input": items, "tools": tools}
if request.instructions:
body["instructions"] = request.instructions
if choice == "none":
body["tool_choice"] = "none"
prompt = self.prepare_native_prompt(
chat_request, protocol="responses", body=body
)
tools = prompt.tools
request._tool_state = msgspec.structs.replace(
request._tool_state,
chat_tools=tools,
names={
call_name: {"namespace": namespace, "name": name}
for call_name, (namespace, name) in prompt.namespace_names.items()
},
discovered_custom_names=frozenset(
tool["name"]
for item in body["input"]
if isinstance(item, dict) and item.get("type") == "tool_search_output"
for tool in item["tools"]
if tool["type"] == "custom"
),
)
if isinstance(choice, dict):
name = choice["name"]
if "namespace" in choice:
# `tools` keeps conversion order, declared before discovered, so
# the last match is the member's latest definition.
member = (choice["namespace"], name)
name = next(
(
tool.function.name
for tool in reversed(tools)
if prompt.namespace_names.get(tool.function.name) == member
),
None,
)
if not any(tool.function.name == name for tool in tools) or (
choice["type"] == "custom"
and name not in self._custom_tool_names(request)
):
raise ResponseToolChoiceError(
f"Tool {choice['name']!r} is not declared in tools without "
"defer_loading or loaded by a tool_search_output"
)
chat_request.tool_choice = ToolChoice(
function=ToolChoiceFuncName(name=name)
)
elif choice == "required" and not tools:
raise ResponseToolChoiceError(
'tool_choice="required" needs a tool the model can call'
)
else:
chat_request.tool_choice = choice if tools else "none"
@classmethod
def _native_input_item(cls, item: Any) -> Any:
if isinstance(item, BaseModel):
item = item.model_dump(exclude_none=True)
if isinstance(item, dict) and item.get("type") == "reasoning":
# recipe reads only `content`; keep the text SGLang replays from a
# summary or a store=false client's state.
message = cls._normalize_response_message_for_chat(item)
if message is None:
return None
return {
"type": "reasoning",
"content": [
{"type": "reasoning_text", "text": message["reasoning_content"]}
],
}
return item
@staticmethod
def _record_called_name(request: ResponsesRequest, call_id: str, name: str) -> None:
"""Keep the name the model called a namespace member by: the item reports
its namespace and own name, which several call names can share."""
if request._tool_state.chat_tools is not None and (
name in request._tool_state.names
):
request._called_tool_names[call_id] = name
@staticmethod
def _replayed_output_items(request: ResponsesRequest, output: list) -> list[dict]:
"""`output` as the history a later request replays: a namespace member's
call under the name the model called it by."""
items = []
for item in output:
item = item.model_dump(exclude_none=True)
called_name = request._called_tool_names.get(item.get("call_id"))
if called_name is not None:
item = {**item, "name": called_name}
del item["namespace"]
items.append(item)
return items
@staticmethod
def _custom_tool_names(request: ResponsesRequest) -> set[str]:
declared = custom_tool_names(request.tools)
if request._tool_state.chat_tools is not None:
# A deferred declaration counts once a search result has loaded it.
declared -= {tool.name for tool in request.tools if tool.defer_loading}
return declared | request._tool_state.discovered_custom_names
@staticmethod
def _client_tool_search(request: ResponsesRequest) -> bool:
"""Whether a `tool_search` call is the client-executed search tool."""
return request._tool_state.chat_tools is not None and any(
tool.type == TOOL_SEARCH for tool in request.tools
)
def _response_tools_to_chat_tools(self, request: ResponsesRequest) -> list[Tool]:
if request._tool_state.chat_tools is not None:
return list(request._tool_state.chat_tools)
# ``function`` and ``custom`` tools flow to chat; built-ins go through
# harmony. A custom tool is shimmed into a single-string function tool.
chat_tools = []
@@ -2133,6 +2487,7 @@ class OpenAIServingResponses(OpenAIServingChat):
tool_follows = opens_item and item_type in (
"function_call",
"custom_tool_call",
"tool_search_call",
)
terminal = event_type in (
"response.completed",
@@ -2215,14 +2570,19 @@ class OpenAIServingResponses(OpenAIServingChat):
)
chat_tools = self._response_tools_to_chat_tools(request)
custom_names = custom_tool_names(request.tools)
custom_names = self._custom_tool_names(request)
namespace_names = (
namespace_function_names(request.tools)
if self.tool_call_parser == "muse"
else {}
else {
call_name: (entry.get("namespace"), entry.get("name", call_name))
for call_name, entry in request._tool_state.names.items()
if entry.get("namespace")
}
)
tool_choice = request.effective_tool_choice()
required_json = self._uses_required_tool_json(request, chat_tools)
client_tool_search = self._client_tool_search(request)
encoding_output = self._encoding_output()
token_stream = (
encoding_output.new_responses_stream(
@@ -2453,7 +2813,13 @@ class OpenAIServingResponses(OpenAIServingChat):
return []
arguments = state["arguments"]
events: list = []
if state["custom"]:
done_event = openai_responses_types.ResponseOutputItemDoneEvent
if state["search"]:
completed_item = _tool_search_call(
state["item_id"], state["call_id"], arguments
)
done_event = ResponseToolSearchCallEvent
elif state["custom"]:
payload = decode_custom_tool_input(arguments)
# Deltas cannot be retracted, so a payload that no longer
# extends what was already streamed defers to the streamed text.
@@ -2520,7 +2886,7 @@ class OpenAIServingResponses(OpenAIServingChat):
)
events.append(
_send_event(
openai_responses_types.ResponseOutputItemDoneEvent(
done_event(
type="response.output_item.done",
sequence_number=-1,
output_index=state["output_index"],
@@ -2717,27 +3083,43 @@ class OpenAIServingResponses(OpenAIServingChat):
current_output_index += 1
name = call.name or ""
is_custom = name in custom_names
is_search = client_tool_search and name == TOOL_SEARCH
namespace, name = namespace_names.get(name, (None, name))
prefix = (
"ctc" if is_custom else "tsc" if is_search else "fc"
)
state = {
"item_id": (
f"ctc_{random_uuid()[:8]}"
if is_custom
else f"fc_{random_uuid()[:8]}"
),
"item_id": f"{prefix}_{random_uuid()[:8]}",
"call_id": f"call_{random_uuid()[:24]}",
"output_index": current_output_index,
"name": name,
"namespace": namespace,
"arguments": "",
"custom": is_custom,
"search": is_search,
"payload": "",
"added": False,
"done": False,
}
tool_call_states[tool_index] = state
self._record_called_name(
request, state["call_id"], call.name or ""
)
if not state["added"]:
state["added"] = True
if state["custom"]:
added_event = (
openai_responses_types.ResponseOutputItemAddedEvent
)
if state["search"]:
# Search arguments are reported whole when it completes.
added_item = ResponseToolSearchCall(
id=state["item_id"],
call_id=state["call_id"],
arguments={},
status="in_progress",
)
added_event = ResponseToolSearchCallEvent
elif state["custom"]:
added_item = ResponseCustomToolCall(
type="custom_tool_call",
id=state["item_id"],
@@ -2756,7 +3138,7 @@ class OpenAIServingResponses(OpenAIServingChat):
**_namespace_field(state),
)
yield _send_event(
openai_responses_types.ResponseOutputItemAddedEvent(
added_event(
type="response.output_item.added",
sequence_number=-1,
output_index=state["output_index"],
@@ -2786,7 +3168,7 @@ class OpenAIServingResponses(OpenAIServingChat):
delta=delta,
)
)
else:
elif not state["search"]:
yield _send_event(
openai_responses_types.ResponseFunctionCallArgumentsDeltaEvent(
type="response.function_call_arguments.delta",
@@ -2936,10 +3318,7 @@ class OpenAIServingResponses(OpenAIServingChat):
self.response_store[final_response.id] = final_response
self.msg_store[final_response.id] = self._response_input_history(
request
) + [
item.model_dump(exclude_none=True)
for item in final_response.output
]
) + self._replayed_output_items(request, final_response.output)
response_dict = _sanitize_response_dict(final_response.model_dump())
@@ -5,6 +5,7 @@ parses them, and tool-call arguments arrive as the API streams them."""
from __future__ import annotations
import json
import logging
from collections.abc import Sequence
from typing import Any, Optional
@@ -17,6 +18,11 @@ from sglang.srt.parser.deepseek_recipe_renderer import (
load_deepseek_recipe,
)
logger = logging.getLogger(__name__)
# The most tokens one UTF-8 character can span, a byte each.
_MAX_UTF8_BYTES = 4
class RecipeToolCallDelta(msgspec.Struct, frozen=True):
"""A piece of one call's JSON arguments; the first piece also names the call."""
@@ -74,6 +80,7 @@ class DeepSeekRecipeOutputParser:
separate_reasoning: bool = True,
stream_reasoning: bool = True,
stop: Sequence[str] = (),
callable_names: Optional[frozenset[str]] = None,
):
recipe = load_deepseek_recipe()
options = recipe.ParsingOptions(
@@ -101,6 +108,10 @@ class DeepSeekRecipeOutputParser:
# out, so a matched stop is cut before recipe has emitted any of it.
self._stop_hold_chars = max(map(len, stop), default=1) - 1
self._pending_ids: list[int] = []
self._callable_names = callable_names
# recipe's index of each call so far, mapped to the index it is reported
# at; None for a dropped call.
self._call_indices: dict[int, Optional[int]] = {}
self.num_consumed_tokens = 0
def feed(self, token_ids: Sequence[int]) -> RecipeOutputDelta:
@@ -192,6 +203,27 @@ class DeepSeekRecipeOutputParser:
+ self._hf_tokenizer.encode(text[:cut], add_special_tokens=False).ids
)
def _reported_call(
self, call: RecipeToolCallDelta
) -> Optional[RecipeToolCallDelta]:
"""`call` at the index it is reported at; None when it belongs to a call
outside `callable_names`."""
if self._callable_names is None:
return call
if call.name is not None:
index = None
if call.name in self._callable_names:
index = sum(kept is not None for kept in self._call_indices.values())
else:
logger.warning(
"Model attempted to call undefined function: %s", call.name
)
self._call_indices[call.index] = index
index = self._call_indices[call.index]
if index is None:
return None
return msgspec.structs.replace(call, index=index)
def _deltas(self, chunks: list[Any]) -> RecipeOutputDelta:
delta = RecipeOutputDelta()
for chunk in chunks:
@@ -202,12 +234,19 @@ class DeepSeekRecipeOutputParser:
reasoning = piece.get("reasoning_content") or ""
content = piece.get("content") or ""
tool_calls = tuple(
RecipeToolCallDelta(
index=call["index"],
arguments=call["function"]["arguments"],
name=call["function"].get("name"),
call
for call in map(
self._reported_call,
(
RecipeToolCallDelta(
index=call["index"],
arguments=call["function"]["arguments"],
name=call["function"].get("name"),
)
for call in piece.get("tool_calls") or ()
),
)
for call in piece.get("tool_calls") or ()
if call is not None
)
if self._reasoning_as_content:
content, reasoning = reasoning + content, ""
@@ -6,6 +6,7 @@ from __future__ import annotations
import copy
import functools
import json
import logging
from types import ModuleType
from typing import Any, Optional, Union
@@ -15,6 +16,8 @@ DEEPSEEK_RECIPE_INSTALL_HINT = (
"`pip install deepseek-recipe`."
)
logger = logging.getLogger(__name__)
MODEL_NAMES = {"dsv4": "DeepSeek-V4-Pro", "dsv41": "DeepSeek-V4.1-Flash"}
# The checkpoint tokenizer's image token; recipe's bundled tokenizer spells it <|image|>.
IMAGE_TOKEN = "<|deepseek_image|>"
@@ -95,6 +98,23 @@ def pre_normalize(body: dict[str, Any]) -> dict[str, Any]:
return request
def loaded_tools(conversation: Any) -> list[Any]:
"""The tools the model can call after `conversation`'s history: those it does
not defer and the deferred ones a system message has loaded."""
recipe = load_deepseek_recipe()
loaded = {
tool.name
for message in conversation.messages
if isinstance(message, recipe.SystemMessage)
for tool in message.tools or []
}
return [
tool
for tool in conversation.tools
if not tool.defer_loading or tool.name in loaded
]
def _error_message(body: Any) -> str:
try:
return json.loads(body)["error"]["message"]
@@ -110,6 +130,20 @@ class DeepSeekRecipe:
recipe = load_deepseek_recipe()
self.model = MODEL_NAMES[chat_encoding_spec]
self.hf_tokenizer = tokenizer
self._fast_prompt_tokenizer = None
if type(tokenizer).__module__.partition(".")[0] == "fastokens":
from fastokens import Tokenizer
if hasattr(Tokenizer, "encode_segments"):
self._fast_prompt_tokenizer = Tokenizer.from_json_str(
tokenizer.to_str()
)
logger.info("DeepSeek recipe prompt encoder: fastokens segmented")
else:
logger.warning(
"deepseek-recipe prompt acceleration requires fastokens.encode_segments; "
"using the reference prompt tokenizer"
)
self.special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
self._options = recipe.ConversionOptions().with_special_tokens(
self.special_tokens
@@ -146,6 +180,19 @@ class DeepSeekRecipe:
) -> list[int]:
"""Prompt ids for `body` (see `pre_normalize`); a request recipe rejects
raises ValueError. `image_urls` receives the images in placeholder order."""
conversation = self.convert(body, effort_budget=effort_budget)
prompt_ids = self.encode_conversation(conversation, effort_budget=effort_budget)
if image_urls is not None:
image_urls.extend(self.image_urls(conversation))
if task is not None:
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
return prompt_ids
def convert(
self, body: dict[str, Any], *, effort_budget: Optional[int] = None
) -> Any:
"""Convert `body` to recipe's `Conversation`; invalid requests raise ValueError.
Pass the same `effort_budget` to `encode_conversation`."""
recipe = load_deepseek_recipe()
normalized = pre_normalize(body)
normalized["model"] = self.model
@@ -155,16 +202,104 @@ class DeepSeekRecipe:
converted = recipe.ChatCompletionRequest(normalized).convert(self._options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
prompt_ids = self._encoding.encode(converted.conversation)
if image_urls is not None:
rendered = self._encoding.render_conversation(converted.conversation)
image_urls.extend(
source.url or source.data_url for source in rendered.image_sources
)
if effort_budget is not None and body["thinking"]:
return converted.conversation
def convert_native(
self, protocol: str, body: dict[str, Any], *, thinking: bool
) -> Any:
"""Convert a Responses or Messages request to recipe's `Conversation` in the
host's thinking mode; invalid requests raise ValueError."""
recipe = load_deepseek_recipe()
request_type = {
"responses": recipe.ResponsesRequest,
"messages": recipe.MessagesRequest,
}[protocol]
options = self._options.with_default_thinking_mode(thinking)
try:
converted = request_type({**body, "model": self.model}).convert(options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
return converted.conversation
def with_host_settings(
self,
conversation: Any,
*,
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> Any:
"""`conversation` with what recipe's Responses and Messages conversions
leave out: a JSON response schema, and SGLang's reading of a trailing
assistant turn, continued or else re-sent as a user turn."""
recipe = load_deepseek_recipe()
messages = conversation.messages
last = messages[-1] if messages else None
if last is not None and last.role == "assistant":
# recipe closes the turn and opens another; SGLang continues it or,
# as `_handle_last_assistant_message`, re-sends it as a user turn.
if continue_final_message:
last = recipe.AssistantMessage(
last.content, last.reasoning_content, prefix=True
)
else:
last = recipe.UserMessage(last.content)
messages = [*messages[:-1], last]
elif response_schema is None:
return conversation
return recipe.Conversation(
messages,
thinking_mode=conversation.thinking_mode,
tools=conversation.tools,
tool_choice=conversation.tool_choice,
reasoning_effort=conversation.reasoning_effort,
response_schema=(
response_schema
if response_schema is not None
else conversation.response_schema
),
)
def namespace_tool_names(
self, namespaces: list[dict[str, Any]]
) -> dict[str, tuple[str, str]]:
"""The name recipe's Responses conversion calls each member of `namespaces`
by, mapped to its namespace and its name there. `namespaces` lists the
request's namespace tools in conversion order: declared, then discovered."""
recipe = load_deepseek_recipe()
body = {"model": self.model, "input": "", "tools": namespaces}
try:
return recipe.ResponsesRequest(body).namespace_tool_map(self._options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
def image_urls(self, conversation: Any) -> list[str]:
"""The images of `conversation` in the order its prompt places them."""
rendered = self._encoding.render_conversation(conversation)
return [source.url or source.data_url for source in rendered.image_sources]
def encode_conversation(
self, conversation: Any, *, effort_budget: Optional[int] = None
) -> list[int]:
"""Encode a recipe `Conversation`, including a prefix of its messages."""
if self._fast_prompt_tokenizer is None:
prompt_ids = self._encoding.encode(conversation)
else:
rendered = self._encoding.render_conversation(conversation)
prompt = rendered.prompt
segments = []
cursor = 0
# Recipe distinguishes control tokens from identical spellings in
# user text. Its Python ranges use code-point offsets, not bytes.
for start, end in rendered.special_ranges:
if start > cursor:
segments.append((prompt[cursor:start], False))
segments.append((prompt[start:end], True))
cursor = end
if cursor < len(prompt):
segments.append((prompt[cursor:], False))
prompt_ids = self._fast_prompt_tokenizer.encode_segments(segments).ids
if effort_budget is not None and conversation.thinking_mode:
prompt_ids = self._replace_effort_budget(prompt_ids, effort_budget)
if task is not None:
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
return prompt_ids
def parse_start(
@@ -97,7 +97,9 @@ class AnthropicMessagesMixin:
)
serving = AnthropicServing(
openai_serving_chat=SimpleNamespace(supports_inline_system=True)
openai_serving_chat=SimpleNamespace(
supports_inline_system=True, native_tool_loading=False
)
)
chat_request = serving._convert_to_chat_completion_request(anthropic_request)
converted = chat_request.model_dump()
@@ -31,6 +31,7 @@ register_cpu_ci(est_time=11, suite="base-a-test-cpu")
class _FakeOpenAIServingChat:
native_reasoning_history = False
native_tool_loading = False
def __init__(self, stream_lines=None, chat_template=None):
self.stream_lines = stream_lines or []
@@ -61,6 +62,7 @@ class _FakeNonStreamingErrorOpenAI:
"""Returns an error response or raises an exception from the OpenAI handler."""
supports_inline_system = False
native_tool_loading = False
def __init__(self, status_code=400, body=None, content=None, exception=None):
self._status_code = status_code
@@ -496,6 +498,26 @@ class TestAnthropicServing(unittest.TestCase):
],
)
def test_tool_addition_fails_where_tools_cannot_load_mid_conversation(self):
"""A tool_addition block must not vanish on a model that cannot render it."""
request = self._anthropic_request(
messages=[
{"role": "user", "content": "hello"},
{
"role": "system",
"content": [
{
"type": "tool_addition",
"tool": {"type": "tool_reference", "name": "Bash"},
}
],
},
]
)
with self.assertRaisesRegex(ValueError, "tool_addition"):
self._serving()._convert_to_chat_completion_request(request)
def test_builtin_web_search_tool_without_schema_is_skipped(self):
request = AnthropicMessagesRequest.model_validate(
{
File diff suppressed because it is too large Load Diff
@@ -204,6 +204,25 @@ class TestDeepSeekRecipe(CustomTestCase):
}
)
def test_fastokens_preserves_prompt_ids(self):
try:
from fastokens import Tokenizer, _ConfiguredTokenizerShim
except ImportError:
self.skipTest("fastokens is not installed")
if not hasattr(Tokenizer, "encode_segments"):
self.skipTest("fastokens has no segmented encoder")
candidate = renderer.DeepSeekRecipe(
"dsv41", _ConfiguredTokenizerShim(self.tokenizer)
)
self.assertIsNotNone(candidate._fast_prompt_tokenizer)
message = {"role": "user", "content": "中文 👩🏽‍💻 <think>x</think>"}
for thinking in (False, True):
body = {"messages": [message], "thinking": thinking}
with self.subTest(thinking=thinking):
self.assertEqual(
candidate.encode(body), self.deepseek_recipe.encode(body)
)
if __name__ == "__main__":
unittest.main()