Compare commits
10
Commits
main
..
b300/dynamo
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aa23c29d52 | ||
|
|
99603451b9 | ||
|
|
81428f230c | ||
|
|
2372edd73d | ||
|
|
34d2224b7f | ||
|
|
36f6a9cdb3 | ||
|
|
2b75283daa | ||
|
|
0fbd87d739 | ||
|
|
bcfdd8a1f7 | ||
|
|
6f546386ca |
@@ -0,0 +1,41 @@
|
||||
# Dynamo-enabled B300 serving image.
|
||||
#
|
||||
# The sglang runtime tree inside the base image is byte-identical to
|
||||
# b300/dynamo @ 8b67636010 (the only commits between the baked tree
|
||||
# 433257c7b0 and 8b67636010 are deploy/docs), so this layer only swaps the
|
||||
# preinstalled Dynamo 1.6.0.dev snapshot for the released 1.6.2 runtime.
|
||||
#
|
||||
# Build:
|
||||
# docker build -t ymkymx/sglang:dynamo-8b6763601-baked-20261117 .
|
||||
# or use ./build-dynamo-image.sh (also imports into containerd).
|
||||
|
||||
ARG BASE=ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
|
||||
FROM ${BASE}
|
||||
|
||||
ARG DYNAMO_VERSION=1.6.2
|
||||
ARG PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
|
||||
LABEL ai.sglang.build.commit="8b67636010e54fc96e5aeba9b0139b38aa18b000"
|
||||
LABEL ai.dynamo.version="${DYNAMO_VERSION}"
|
||||
LABEL org.opencontainers.image.description="SGLang B300 (b300/dynamo) + NVIDIA Dynamo ${DYNAMO_VERSION}"
|
||||
|
||||
# --no-deps: the base image already carries the full runtime dependency set
|
||||
# (torch 2.14.1, ai-dynamo deps, nixl 1.5.0). Resolving would try to pull
|
||||
# sglang==0.5.19 from the `sglang` extra and downgrade torch.
|
||||
RUN pip uninstall -y ai-dynamo ai-dynamo-runtime >/dev/null 2>&1 || true \
|
||||
&& pip install --no-cache-dir --no-deps -i "${PIP_INDEX}" \
|
||||
"ai-dynamo==${DYNAMO_VERSION}" "ai-dynamo-runtime==${DYNAMO_VERSION}"
|
||||
|
||||
# CPU-only smoke: both CLIs must import the sglang args we build from source.
|
||||
RUN python - <<'PY'
|
||||
import importlib.metadata as m
|
||||
import sglang
|
||||
import dynamo.sglang # noqa: F401
|
||||
import dynamo.frontend # noqa: F401
|
||||
print("dynamo", m.version("ai-dynamo"), "runtime", m.version("ai-dynamo-runtime"))
|
||||
print("sglang", sglang.__file__)
|
||||
PY
|
||||
RUN python -m dynamo.sglang --help >/dev/null && python -m dynamo.frontend --help >/dev/null
|
||||
|
||||
ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
|
||||
CMD []
|
||||
@@ -0,0 +1,216 @@
|
||||
# Dynamo 支持(b300/dynamo)
|
||||
|
||||
> 分支:`b300/dynamo`(基点 `8b67636010`,= `main` = `b300/integration` 的 2026-11-17 c580 基线)
|
||||
> 状态:**代码/镜像/启动脚本已就绪,静态兼容性已验证;GPU 运行时验证待窗口**
|
||||
> (2026-11-17 时 `ds41-gist-exp` 占满 8 卡)。
|
||||
|
||||
本目录是 B300 线切换到 **NVIDIA Dynamo** 的入口:Dynamo frontend + 我们的 sglang
|
||||
worker(P/D 分离)。它与现有 `ds41-gist-exp` 的区别:**入口 + PD 编排从 g9i gateway
|
||||
换成 Dynamo frontend(`:33000`)**;**KV 数据面仍是 sglang 原生 disaggregation(mooncake)**。
|
||||
|
||||
## 1. 兼容性结论(2026-11-17)
|
||||
|
||||
- **不需要 sglang 代码改动**。Dynamo 1.6.2 的 `dynamo.sglang` 后端通过 `sgl.Engine`
|
||||
公开接口工作,`_compat.py` 对 0.5.18/0.5.19 的兼容层在当前树(main-era)全部命中
|
||||
新版分支。
|
||||
- 用 Dynamo 1.6.2 自带单测跑了当前树(`PYTHONPATH=<tree>/python`,镜像内先装
|
||||
`pytest-asyncio`):**985 passed / 20 failed / 1 skipped**。同一镜像用自带 sglang
|
||||
nightly 跑对照也是 **985 passed / 20 failed / 1 skipped**,`diff` 后失败用例集合
|
||||
**逐条相同**(均为视频生成等既有环境相关用例)。→ 无 API / 行为漂移。
|
||||
- `python -m dynamo.sglang --help`(1398 行,sglang 参数已合并)与
|
||||
`python -m dynamo.frontend --help` 均 rc=0。
|
||||
- Dynamo 依赖的 `ServerArgs` 字段(`enable_forward_pass_metrics`、
|
||||
`forward_pass_metrics_ipc_name`、`elastic_ep_backend`、`dllm_algorithm`、
|
||||
`incremental_streaming_output`、`kv_events_config`、`allow_auto_truncate` 等)
|
||||
在当前树全部存在。
|
||||
|
||||
复现命令(CPU,容器内):
|
||||
|
||||
```bash
|
||||
docker run --rm --entrypoint bash \
|
||||
-e PYTHONPATH=/data/ymk/wt-dynamo/python \
|
||||
-v /data/ymk/wt-dynamo:/data/ymk/wt-dynamo:ro \
|
||||
<dynamo-image> -c 'python -m dynamo.sglang --help >/dev/null && echo ok'
|
||||
```
|
||||
|
||||
## 2. 版本矩阵
|
||||
|
||||
| 组件 | 版本 | 备注 |
|
||||
|---|---|---|
|
||||
| sglang | `b300/dynamo` @ `8b67636010` | 与镜像内已烘源码 `433257c7b0` 的 `python/sglang` **逐字节一致**(中间只有 deploy/docs 提交) |
|
||||
| Dynamo | **1.6.2**(`ai-dynamo` + `ai-dynamo-runtime`) | 基底镜像原本带 `1.6.0.dev20261019`,Dockerfile 覆盖 |
|
||||
| nixl | 1.5.0(`nixl-cu13` 1.5.0) | 基底已带;Dynamo 的 SGLang 后端**不用**它做 KV 传输 |
|
||||
| torch | 2.14.1+cu130 | 基底已带,不因 Dynamo 变化 |
|
||||
| sglang-kernel | 0.4.10 | 基底已带,与当前树 pin 一致 |
|
||||
| mooncake | 0.3.14(L3 SSD) | 基底已带(见 `../README.md`) |
|
||||
|
||||
- KV 传输仍是 **sglang 原生 disaggregation + mooncake**;`--connector nixl` 只对
|
||||
vLLM/TRT-LLM 生效,对 SGLang 是 no-op(10-26 已确认,本脚本不带它)。
|
||||
- NVLink:`MC_INTRANODE_NVLINK=true` + `MC_INTRA_NVLINK=true` + `--pid host` +
|
||||
宿主已烘的 NVLink host 修复。
|
||||
|
||||
### 2.1 PD 的两层要分开看(常被混淆)
|
||||
|
||||
| 层 | 由谁实现 | 在本部署里的表现 |
|
||||
|---|---|---|
|
||||
| **PD 编排 / 路由**:选 P、选 D、把 `bootstrap_info` 从 P 传给 D、流式回传 | **Dynamo 自己的 Rust runtime + frontend PrefillRouter**(不是 sglang-router/mini_lb) | `dynamo.frontend :33000`;Dynamo 的 `sglang` handler 只负责把 bootstrap 信息塞进/取出 `Engine.async_generate` |
|
||||
| **KV 数据面**:bootstrap 握手后的 KV 传输 | **sglang 原生 disaggregation**(`--disaggregation-mode` + `--disaggregation-transfer-backend mooncake`) | P/D 日志的 `Using Intra-Node NVLink transport`;10-26 的 NVLink host 补丁仍然生效 |
|
||||
|
||||
Dynamo 的 `--connector nixl` / `--kv-transfer-config` 只对 vLLM/TRT-LLM 生效(vLLM 需要
|
||||
`--kv-transfer-config '{"kv_connector":"NixlConnector",...}'`);SGLang 后端把
|
||||
`bootstrap_host/port/room` 直接传给 `Engine.async_generate`,传输由 sglang 的 bootstrap
|
||||
server + mooncake 完成(`prefill_handler.py` 产出 `disaggregated_params`,
|
||||
`decode_handler.py` 消费 `bootstrap_info`)。所以 P/D 两侧仍然必须写
|
||||
`--disaggregation-mode`,且 `--disaggregation-transfer-backend` 要与 sglang 侧一致。
|
||||
|
||||
> 结论:**编排是 Dynamo 的,KV 线是 sglang 的**。两者不是同一套,但也不是各走各的——
|
||||
> Dynamo 控制面负责“谁和谁配对”,sglang 数据面负责真正搬 KV。
|
||||
|
||||
### 2.3 Dynamo 1.6.2 的 tokenizer / chat processor(实测)
|
||||
|
||||
DeepSeek-V4.1 没有 HF `chat_template`,默认的 Rust processor 无法渲染;需要用
|
||||
SGLang 的 pre/post processor:
|
||||
|
||||
```
|
||||
--dyn-chat-processor sglang \
|
||||
--dyn-preprocess-workers 8 \
|
||||
--tokenizer fastokens --trust-remote-code \
|
||||
--tool-call-parser deepseekv41 \
|
||||
--reasoning-parser deepseek-v41
|
||||
```
|
||||
|
||||
本次踩通/修掉的四个点:
|
||||
|
||||
1. **缺 `enencoding_dsv4`**:`--dyn-chat-processor sglang` 会 import
|
||||
`sglang.srt.entrypoints.openai.enencoding_dsv4`;我们的 fork 用 deepseek-recipe,没有该模块,
|
||||
chat 直接 500。已在 `b300/dynamo` 加 shim `enencoding_dsv4.py`(基于 recipe 的
|
||||
`render_conversation`),token ids 与生产 `DeepSeekRecipe.encode` 逐 token 相同。
|
||||
2. **`--tokenizer fastokens` 不会自动传给 SGLang processor**:该 flag 只作用于 Dynamo 的
|
||||
Rust model-card tokenizer;`sglang_processor._load_tokenizer()` 调 `get_tokenizer()`
|
||||
时默认 `huggingface`。需要把 `FrontendConfig.tokenizer_backend` 传进去(FE hostPath 补丁),
|
||||
日志出现 `fastokens backend enabled - transformers patched successfully` 才算生效。
|
||||
3. **v41 thinking 解析**:Dynamo 的 `_THINKING_OPT_IN` 缺 `deepseek-v41`,显式
|
||||
`chat_template_kwargs.thinking=true` 时 `reasoning_content` 不会拆出来。已补上;
|
||||
默认请求仍是 chat(worker 没有发布 `default_thinking_mode`;生产默认 thinking 需要
|
||||
register 侧补丁,TODO)。
|
||||
4. **`--gateway-workers N`(worker 侧)** 是另一条路:给默认 Rust processor 用,启动 N 个
|
||||
tokenizer/gateway 进程并设置 SGLang `--tokenizer-worker-num`;`--use-sglang-tokenizer`
|
||||
已 deprecated。本次走 frontend processor,不需要它。
|
||||
|
||||
另:`encoding_dsv4`/`thinking` 的 FE 补丁目前以 hostPath 挂载在容器里,尚未烘进镜像;
|
||||
重建镜像时要一起带上(见 `start_dynamo_pd.sh` 的 `-v` 列表)。
|
||||
|
||||
### 2.4 2026-11-17 首次运行结果(4P(dp4)+4D(dp4))
|
||||
|
||||
- 启动约 7 分钟;P `max_total_num_tokens=16,157,696`、D `58,859,520`(与 11-17
|
||||
`ds41-gist-exp` 基线一致);P/D 均出现 `Using Intra-Node NVLink transport`。
|
||||
- chat / tool_calls / reasoning_content 均可;prefix cache 正常(16k prompt 第二轮
|
||||
cached 15,872;80k 第二轮 cached 79,872)。
|
||||
- 32 并发 × ~900 token prompt(85% 命中,warm):**wall 0.62 s,32/32 成功**;
|
||||
16k 冷 0.47 s / 热 0.24 s;80k 热 0.25 s(冷 80k ~1.04 s,HF tokenizer 时的数字)。
|
||||
**重启后前 1–2 波会明显慢(8–9 s)是 warmup 伪影,benchmark 必须先热身。**
|
||||
- P 的 OTel batch-span link 会报 `expected bytes, TraceNullContext found`(旧镜像);
|
||||
已在分支修 `batch_telemetry.py`,本次运行用 `SGLANG_BATCH_SPAN=0` 规避。
|
||||
|
||||
### 2.2 Dynamo 不会消除 sglang 的 bootstrap / 握手尾巴
|
||||
|
||||
Dynamo 换掉的是控制面,sglang 内部的 bootstrap 路径原样保留:
|
||||
|
||||
- P 侧 `PrefillBootstrapQueue.pop_bootstrapped()` 与 `resolve_waiting_queue_bootstrap()`
|
||||
仍然在 scheduler 每个 pass 里做 `poll_and_all_reduce_attn_cp_tp_group(...)`
|
||||
(`disaggregation/prefill.py:495-556`、`638-675`);bootstrap 完成只在 pass 边界被发现。
|
||||
- D 侧 `DecodePreallocQueue` / `_ensure_prefill_info` / 传输接收队列也一样。
|
||||
- 因此 `prefill_bootstrap`(P queue entry → bootstrap done)与 `decode_transferred`
|
||||
(等 P 完成 + 收 KV)的口径不变,Dynamo frontend 必须等它们完成才会开始 decode;
|
||||
长 pass / DP skew / 大 batch 会按 pass 放大这个延迟。
|
||||
|
||||
实测参考(当前栈):c620 下 `prefill_bootstrap` p50 0.23–0.36 s、p95 1.66 s(fresh)/ 5.05 s
|
||||
(不重启),而 `mooncake_worker_send_session` p50 只有 2.6 ms —— 卡的是握手/轮询,不是搬 KV。
|
||||
Dynamo 的收益在入口/路由/可观测性;要压 bootstrap 尾部必须改 sglang scheduler 的轮询/预分配路径
|
||||
(或先用 optimistic prefill + `hicache-write-policy write_back` 把 prefill 与握手重叠)。
|
||||
|
||||
## 3. 构建镜像
|
||||
|
||||
```bash
|
||||
cd deploy/b300-k8s/dynamo
|
||||
./build-dynamo-image.sh # 构建 + ctr import
|
||||
# IMPORT=0 只构建;TAG=... 自定义 tag
|
||||
```
|
||||
|
||||
默认产物:`ymkymx/sglang:dynamo-8b6763601-baked-20261117`
|
||||
(= `sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:...`,同一 image ID)。
|
||||
|
||||
要点:Dynamo 必须 `pip install --no-deps`。基底已带全部运行时依赖;一旦让 pip 解析,
|
||||
`ai-dynamo` 的 `sglang` extra 会试图装 `sglang==0.5.19` 并降级 torch。
|
||||
|
||||
## 4. 启动(docker,4P(dp4)+4D(dp4))
|
||||
|
||||
先停掉占卡的工作负载并确认 GPU 0-7 为 0 MiB,再:
|
||||
|
||||
```bash
|
||||
./start_dynamo_pd.sh
|
||||
./smoke.sh # /v1/models + /v1/chat/completions
|
||||
tail -f /data/ymk/gist-dsv41/dyn-{fe,p,d}.log
|
||||
```
|
||||
|
||||
端口:frontend **33000**、P **30000**、D **30300**;P/D 的 sglang 指标仍在原端口,
|
||||
Dynamo 自己的指标在 frontend。
|
||||
|
||||
> ⚠️ 启动耗时与普通 P/D 栈相同(P/D 各 ~6-10 分钟;`/root/.cache/sglang` 挂载可复用
|
||||
> autotune/JIT 缓存)。**先等 P/D ready 再开压测**:Dynamo 有冷启动首波长尾
|
||||
> (空闲后首波 3–22s,暖机后消失)。
|
||||
|
||||
## 5. 与 2026-10-26 Dynamo 试验的差异
|
||||
|
||||
| 项 | 10-26 试验 | 本目录 |
|
||||
|---|---|---|
|
||||
| Dynamo | runtime 镜像 `sglang-runtime:1.6.0` | PyPI **1.6.2**(烘进 B300 镜像) |
|
||||
| 模型 | `DeepSeek-V4.1-Flash` | `DeepSeek-V4.1-Flash-1017`(served name 保持原路径) |
|
||||
| 并行 | tp4 + tp4(`--moe-a2a-backend none`) | **dp4 + dp4**(dp-attention + dp-lm-head + megamoe + flashmla_mega) |
|
||||
| 入口 | `dynamo.frontend :33000` | 同上 |
|
||||
| 缓存 | L2/L3 32GB | 当前生产 L3 SSD 4TB + `write_through` |
|
||||
| 门控 | `SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0` | 同(依赖树内 `7f6fe22445`/`ee9170e7bb` fail-soft) |
|
||||
|
||||
dp4 是 DSpark PD 的硬要求:两侧 `attn_tp_size` 必须相等。dp4 下 attn_tp = tp/dp = 1,
|
||||
P/D 一致;若一侧 tp4、另一侧 dp4,`disaggregation/common/conn.py:1355` 会直接拒绝
|
||||
(`DSpark PD requires the same TP size on both servers`)。
|
||||
|
||||
## 6. 已知约束 / 坑
|
||||
|
||||
1. **`--host 0.0.0.0` 必须**:sglang bootstrap server 跟随 `--host`;写 127.0.0.1 时
|
||||
D 侧拿不到 prefill parallel info → `HTTP 500: Decode handshake failed`。
|
||||
2. **`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python`**:baked 镜像没有
|
||||
`rust_tree_core` `.so`(fake-cargo 阻止现场编译);1M 上下文下 unified cache 会选
|
||||
Rust TreeCore 并 `ModuleNotFoundError`。本脚本已设为 `python`。
|
||||
3. **Dynamo 1.6.2 安装用 `--no-deps`**:见 §3。`aisimulate` 在清华源只有
|
||||
`0.13.0.dev2`(Dynamo 1.6.2 元数据要 `0.13.0`),实测 import 正常。
|
||||
4. **`--dyn-preprocess-workers`** 取代了旧报告的 “gateway workers”:1.6.2 默认
|
||||
auto(每个 model 最多 2),可显式设 4–8 提升高并发 tokenize 吞吐;仅在
|
||||
`--dyn-chat-processor sglang` 下生效。
|
||||
5. **Dynamo frontend 没有 g9i 的准入闸门**(`--g9i-p-queue-reqs-cap` 等)。Dynamo 自己的
|
||||
flow control(`--router-queue-threshold` / `--load-aware`)在 10-26 实测无效,
|
||||
有效闸门仍是客户端/网关并发上限。
|
||||
6. **P/D 的 health / readiness**:`dynamo.sglang` worker 通过 discovery 注册,不使用
|
||||
sglang 的 `/health` 作为就绪判据;k8s 化时探针应指向 frontend,或改用 dynamo 的
|
||||
health payload(`dynamo.sglang.health_check`)。
|
||||
7. `--enable-trace` 的 OTLP 服务名分别为 `dyn-p` / `dyn-d`(与旧试验一致);
|
||||
Jaeger `:4317` 可用。
|
||||
8. 端口分配遵守 `30000-32767` 固定端口纪律;frontend 33000 与现有栈不冲突。
|
||||
|
||||
## 7. 待办
|
||||
|
||||
- [ ] **GPU 冒烟**:等 `ds41-gist-exp` 停/窗口;验证 PD 全链路、NVLink transport 日志
|
||||
(`Using Intra-Node NVLink transport`)、DSpark accept len、L2/L3 命中。
|
||||
- [ ] **性能基线**:c580/1200s(同 `baselines/20261117-dtok16-fastokens` 口径)对比
|
||||
g9i gateway 栈;关注 Dynamo frontend 的 tokenize 开销与冷启动长尾。
|
||||
- [ ] **k8s cw manifest**:`ds41-dynamo`(frontend + p + d + mooncake-master),
|
||||
探针/服务注册按 Dynamo discovery 设计;本目录先提供 docker 形态。
|
||||
- [ ] 视需要把 namespace / Chat-Completions 工具展平(`b300/feat-recipe-ns` 已进
|
||||
`b300/integration`)在 Dynamo frontend 侧的行为纳入 `preflight/issue-set`。
|
||||
|
||||
## 8. 参考
|
||||
|
||||
- 旧运行记录:`D:\B300\experiments\dsv41-gist-deepinfer\REPORT-DYNAMO-20261026.md`
|
||||
- 启动脚本(旧,tp4):`D:\B300\experiments\dsv41-gist-deepinfer\start_dynamo_pd.sh`
|
||||
- Dynamo 仓库(本地):`D:\B300\dynamo`(1.6.0 源码树,仅作 API 参考;实际运行用 1.6.2 wheel)
|
||||
- L3 SSD 基线:`deploy/b300-k8s/README.md`
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build (and optionally load into containerd) the Dynamo-enabled B300 image.
|
||||
#
|
||||
# ./build-dynamo-image.sh # build + ctr import
|
||||
# IMPORT=0 ./build-dynamo-image.sh # build only
|
||||
# TAG=my-tag ./build-dynamo-image.sh # custom tag
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
BASE=${BASE:-ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115}
|
||||
TAG=${TAG:-dynamo-8b6763601-baked-20261117}
|
||||
REPO=${REPO:-sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang}
|
||||
IMPORT=${IMPORT:-1}
|
||||
DYNAMO_VERSION=${DYNAMO_VERSION:-1.6.2}
|
||||
|
||||
echo "base=$BASE tag=$TAG dynamo=$DYNAMO_VERSION"
|
||||
docker build \
|
||||
--build-arg BASE="$BASE" \
|
||||
--build-arg DYNAMO_VERSION="$DYNAMO_VERSION" \
|
||||
-t "ymkymx/sglang:$TAG" \
|
||||
-f Dockerfile .
|
||||
|
||||
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
|
||||
|
||||
docker run --rm --entrypoint bash "ymkymx/sglang:$TAG" -c '
|
||||
python - <<PY
|
||||
import importlib.metadata as m, importlib.util
|
||||
import dynamo.sglang, dynamo.frontend # noqa: F401
|
||||
print("ok: ai-dynamo", m.version("ai-dynamo"), "| sglang", importlib.util.find_spec("sglang").origin)
|
||||
PY
|
||||
python -m dynamo.sglang --help >/dev/null
|
||||
python -m dynamo.frontend --help >/dev/null
|
||||
'
|
||||
|
||||
if [ "$IMPORT" = "1" ]; then
|
||||
echo "importing $REPO:$TAG into containerd (k8s.io) ..."
|
||||
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
|
||||
fi
|
||||
echo "done: $REPO:$TAG"
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dynamo frontend smoke test.
|
||||
# ./smoke.sh [base_url] [model_name]
|
||||
set -euo pipefail
|
||||
BASE=${1:-http://127.0.0.1:33000}
|
||||
MODEL=${2:-/data/models/DeepSeek-V4.1-Flash}
|
||||
|
||||
echo "== /v1/models"
|
||||
curl -fsS "$BASE/v1/models" | head -c 600; echo
|
||||
|
||||
echo "== chat"
|
||||
curl -fsS "$BASE/v1/chat/completions" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"2+3=? Answer with the number only.\"}],\"max_tokens\":64,\"temperature\":0}" \
|
||||
| head -c 1200; echo
|
||||
Executable
+183
@@ -0,0 +1,183 @@
|
||||
#!/usr/bin/env bash
|
||||
# 4P(dp4)+4D(dp4) PD on NVIDIA Dynamo 1.6.2 + our B300 sglang (b300/dynamo @ 8b67636010).
|
||||
#
|
||||
# frontend :33000 (Dynamo router; replaces the g9i gateway)
|
||||
# prefill :30000 GPU 0-3, mooncake NVLink, HiCache L2 + L3 SSD
|
||||
# decode :30300 GPU 4-7
|
||||
#
|
||||
# Differences vs the 2026-10-26 Dynamo bring-up (REPORT-DYNAMO-20261026.md):
|
||||
# * dp4 (tp4 + dp-attention + dp-lm-head) on both sides instead of tp4, matching
|
||||
# the current production stack. Both sides must expose attn_tp_size==1 or the
|
||||
# DSpark PD handshake rejects the pair ("same TP size on both servers").
|
||||
# * image carries Dynamo 1.6.2 (the 10-26 run used the 1.6.0 runtime image).
|
||||
# * L3 SSD mooncake 0.3.14 + the 1017 model / fastokens tokenizer.
|
||||
#
|
||||
# NOT validated on GPU yet (2026-11-17: ds41-gist-exp owns all 8 GPUs).
|
||||
set -uo pipefail
|
||||
|
||||
IMG=${IMG:-ymkymx/sglang:dynamo-8b6763601-baked-20261117}
|
||||
MODEL=${MODEL:-/data/models/DeepSeek-V4.1-Flash-1017}
|
||||
SERVED=${SERVED:-/data/models/DeepSeek-V4.1-Flash}
|
||||
D=${D:-/data/ymk/gist-dsv41}
|
||||
STORE=${STORE:-$D/dynamo-store}
|
||||
L3_CONF=${L3_CONF:-/data/ymk/ds41/mooncake-store-l3ssd.json}
|
||||
JITCACHE=${JITCACHE:-/data/ymk/cache/sglang}
|
||||
FE_PORT=${FE_PORT:-33000}
|
||||
# Frontend tokenizer/chat processing (1.6.2):
|
||||
# --dyn-chat-processor sglang frontend does SGLang-native template rendering,
|
||||
# tokenization, tool/reasoning parsing
|
||||
# --dyn-preprocess-workers N ProcessPool of N tokenizer/template workers
|
||||
# --tokenizer fastokens fastokens BPE encoder (falls back to HF)
|
||||
# Parser names must be explicit: sglang's "auto" is resolved at server startup
|
||||
# and is not understood by the frontend's parser factory.
|
||||
PREPROCESS_WORKERS=${PREPROCESS_WORKERS:-8}
|
||||
TOOL_PARSER=${TOOL_PARSER:-deepseekv41}
|
||||
REASONING_PARSER=${REASONING_PARSER:-deepseek-v41}
|
||||
FE_ARGS=${FE_ARGS:---router-mode round-robin \
|
||||
--dyn-chat-processor sglang \
|
||||
--dyn-preprocess-workers $PREPROCESS_WORKERS \
|
||||
--tokenizer fastokens --trust-remote-code \
|
||||
--tool-call-parser $TOOL_PARSER \
|
||||
--reasoning-parser $REASONING_PARSER}
|
||||
|
||||
mkdir -p "$STORE"; chmod 777 "$STORE" 2>/dev/null || true
|
||||
|
||||
docker rm -f dyn-fe dyn-p dyn-d >/dev/null 2>&1
|
||||
sleep 3
|
||||
# Dynamo/sglang children can outlive the container when --pid host is used.
|
||||
for p in $(ps -eo pid,comm | awk '$2 ~ /^sglang::/ {print $1}'); do kill -9 "$p" 2>/dev/null; done
|
||||
sleep 2
|
||||
|
||||
# L3 store master (not part of the Dynamo graph). Same pod/port as the cw master.
|
||||
if ! docker ps --format '{{.Names}}' | grep -qx dyn-mooncake; then
|
||||
docker rm -f dyn-mooncake >/dev/null 2>&1
|
||||
docker run -d --name dyn-mooncake --network host "$IMG" \
|
||||
/opt/sglang/bin/mooncake_master \
|
||||
--port 50051 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=18081 --metrics_port=19003 \
|
||||
--eviction_high_watermark_ratio=0.90 \
|
||||
--enable_offload=true --offload_on_evict=false \
|
||||
--promotion_on_hit=false --offload_force_evict=false \
|
||||
> "$D/dyn-mooncake.log" 2>&1
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
COMMON_ENVS=(
|
||||
-e SGLANG_RAGGED_VERIFY_MODE=static
|
||||
-e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1
|
||||
-e SGLANG_DSV4_KV_LAYOUT=v41_new
|
||||
-e MC_INTRANODE_NVLINK=true
|
||||
-e MC_INTRA_NVLINK=true
|
||||
-e SGLANG_MOONCAKE_SEND_AUX_TCP=true
|
||||
# The baked image has no rust_tree_core .so; force the Python radix tree.
|
||||
-e SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
|
||||
-e SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600
|
||||
-e SGLANG_DISAGGREGATION_QUEUE_SIZE=16
|
||||
-e SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=32
|
||||
-e SGLANG_BATCH_TELEMETRY=1
|
||||
)
|
||||
P_ENVS=(
|
||||
-e SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1
|
||||
-e SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1
|
||||
-e SGLANG_OPT_DSV41_BF16_INDEX_LOGITS=1
|
||||
-e SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536
|
||||
-e SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0
|
||||
-e SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=$L3_CONF
|
||||
-e SGLANG_HICACHE_PREFETCH_IO_THREADS=4
|
||||
-e SGLANG_HICACHE_BACKUP_IO_THREADS=4
|
||||
-e SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1
|
||||
-e SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH=1
|
||||
# The baked image predates the TraceNullContext guard; keep batch spans off
|
||||
# until the next image build (histograms are unaffected).
|
||||
-e SGLANG_BATCH_SPAN=0
|
||||
)
|
||||
D_ENVS=(
|
||||
-e SGLANG_ENABLE_METRICS_DP_ATTENTION=1
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------- frontend
|
||||
docker run -d --name dyn-fe --network host --entrypoint bash \
|
||||
-v /data:/data -v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/wt-dynamo/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:ro \
|
||||
-v /data/ymk/dynamo-patch-sglang_prepost.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_prepost.py:ro \
|
||||
-v /data/ymk/dynamo-sglang_processor.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_processor.py:ro \
|
||||
"$IMG" \
|
||||
-c "python -m dynamo.frontend --discovery-backend file --http-port $FE_PORT $FE_ARGS \
|
||||
> $D/dyn-fe.log 2>&1"
|
||||
|
||||
# ---------------------------------------------------------------- prefill
|
||||
docker run -d --name dyn-p --gpus all -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
|
||||
"${COMMON_ENVS[@]}" "${P_ENVS[@]}" \
|
||||
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
|
||||
-v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
|
||||
"$IMG" python -m dynamo.sglang \
|
||||
--endpoint dyn://dynamo.prefill.generate --discovery-backend file \
|
||||
--disaggregation-mode prefill \
|
||||
--trust-remote-code --tokenizer-backend fastokens \
|
||||
--model-path "$MODEL" --served-model-name "$SERVED" \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--dyn-default-thinking-mode enabled \
|
||||
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
|
||||
--max-running-requests 512 --tokenizer-worker-num 16 \
|
||||
--load-balance-method total_tokens \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
|
||||
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
|
||||
--cuda-graph-backend-prefill breakable \
|
||||
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
|
||||
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
|
||||
--enable-hierarchical-cache --hicache-ratio 4 \
|
||||
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
|
||||
--hicache-write-policy write_through --hicache-storage-backend mooncake \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
|
||||
--enable-dp-attention-local-control-broadcast \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-p \
|
||||
--watchdog-timeout 3600 \
|
||||
> "$D/dyn-p.log" 2>&1
|
||||
|
||||
# ---------------------------------------------------------------- decode
|
||||
docker run -d --name dyn-d --gpus all -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
|
||||
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
|
||||
"${COMMON_ENVS[@]}" "${D_ENVS[@]}" \
|
||||
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
|
||||
-v "$STORE":/tmp/dynamo_store_kv \
|
||||
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
|
||||
"$IMG" python -m dynamo.sglang \
|
||||
--endpoint dyn://dynamo.backend.generate --discovery-backend file \
|
||||
--disaggregation-mode decode \
|
||||
--trust-remote-code --tokenizer-backend fastokens \
|
||||
--model-path "$MODEL" --served-model-name "$SERVED" \
|
||||
--reasoning-parser auto --tool-call-parser auto \
|
||||
--dyn-default-thinking-mode enabled \
|
||||
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
|
||||
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
|
||||
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
|
||||
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
|
||||
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
|
||||
--mem-fraction-static 0.80 \
|
||||
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
|
||||
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
|
||||
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
|
||||
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
|
||||
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
|
||||
--enable-trace --trace-modules request,mooncake \
|
||||
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-d \
|
||||
--watchdog-timeout 3600 \
|
||||
> "$D/dyn-d.log" 2>&1
|
||||
|
||||
sleep 12
|
||||
docker ps --format '{{.Names}}\t{{.Status}}' | grep -E '^dyn-' || true
|
||||
echo "---- frontend log"; tail -3 "$D/dyn-fe.log" 2>/dev/null
|
||||
echo "---- prefill log"; tail -3 "$D/dyn-p.log" 2>/dev/null
|
||||
echo "---- decode log"; tail -3 "$D/dyn-d.log" 2>/dev/null
|
||||
echo "[$(date +%H:%M:%S)] launched: FE=$FE_PORT P=30000 D=30300 (image $IMG)"
|
||||
@@ -0,0 +1,105 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Compatibility shim for Dynamo's SGLang chat processor.
|
||||
|
||||
Upstream SGLang builds expose ``sglang.srt.entrypoints.openai.encoding_dsv4``
|
||||
with ``encode_messages(messages, *, thinking_mode, reasoning_effort) -> str``.
|
||||
Dynamo's frontend (``dynamo.frontend.sglang_prepost``) imports that module to
|
||||
render DeepSeek-V4/V4.1 prompts when the model ships no HF chat template.
|
||||
|
||||
Our DeepSeek-V4 fork replaced the encoder with the ``deepseek-recipe``
|
||||
renderer (``sglang.srt.parser.deepseek_recipe_renderer``). This module exposes
|
||||
the upstream API on top of the recipe, so ``--dyn-chat-processor sglang``
|
||||
works without installing a second SGLang build.
|
||||
|
||||
The returned string is tokenized by the caller. For the V4.1 checkpoint the
|
||||
resulting ids match ``DeepSeekRecipe.encode()`` (the native serving path)
|
||||
byte-for-byte; the renderer is the same code path, only the tokenizer call
|
||||
moves to the caller.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from typing import Any, Optional
|
||||
|
||||
from sglang.srt.parser.deepseek_recipe_renderer import (
|
||||
IMAGE_TOKEN,
|
||||
MODEL_NAMES,
|
||||
load_deepseek_recipe,
|
||||
pre_normalize,
|
||||
)
|
||||
|
||||
# The Dynamo frontend uses this encoder for the V4 family. Our deployment is
|
||||
# DeepSeek-V4.1-Flash; V4-Pro checkpoints need the ``dsv4`` spec instead.
|
||||
_CHAT_ENCODING_SPEC = "dsv41"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _renderer():
|
||||
recipe = load_deepseek_recipe()
|
||||
special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
|
||||
encoding = recipe.DeepseekV41Encoding().with_special_tokens(special_tokens)
|
||||
options = recipe.ConversionOptions().with_special_tokens(special_tokens)
|
||||
return recipe, special_tokens, encoding, options
|
||||
|
||||
|
||||
def encode_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
*,
|
||||
thinking_mode: str = "chat",
|
||||
reasoning_effort: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Render OpenAI-style ``messages`` to a DeepSeek-V4.1 prompt string.
|
||||
|
||||
Mirrors upstream ``encoding_dsv4.encode_messages``:
|
||||
|
||||
* ``thinking_mode`` is ``"thinking"`` or ``"chat"``;
|
||||
* ``reasoning_effort`` is a named tier (``"high"`` / ``"max"``) or None;
|
||||
* tool schemas are attached to the first system message by the caller
|
||||
(Dynamo's convention), not passed as a separate argument.
|
||||
|
||||
Raises ``ValueError`` for prompt shapes the recipe rejects.
|
||||
"""
|
||||
recipe, _special_tokens, encoding, options = _renderer()
|
||||
|
||||
encoding_messages = copy.deepcopy(messages)
|
||||
tools = None
|
||||
if encoding_messages:
|
||||
tools = encoding_messages[0].pop("tools", None)
|
||||
if not tools:
|
||||
tools = None
|
||||
|
||||
# Dynamo already flattens message content and re-serializes tool-call
|
||||
# arguments; keep this defensive normalization for direct callers.
|
||||
for message in encoding_messages:
|
||||
if message.get("content") is None:
|
||||
message["content"] = ""
|
||||
for call in message.get("tool_calls") or []:
|
||||
function = call.get("function") if isinstance(call, dict) else None
|
||||
if isinstance(function, dict) and isinstance(
|
||||
function.get("arguments"), (dict, list)
|
||||
):
|
||||
function["arguments"] = json.dumps(
|
||||
function["arguments"], ensure_ascii=False
|
||||
)
|
||||
|
||||
body: dict[str, Any] = {
|
||||
"messages": encoding_messages,
|
||||
"thinking": thinking_mode == "thinking",
|
||||
}
|
||||
if tools:
|
||||
body["tools"] = tools
|
||||
if reasoning_effort is not None:
|
||||
body["reasoning_effort"] = reasoning_effort
|
||||
|
||||
normalized = pre_normalize(body)
|
||||
normalized["model"] = MODEL_NAMES[_CHAT_ENCODING_SPEC]
|
||||
try:
|
||||
converted = recipe.ChatCompletionRequest(normalized).convert(options)
|
||||
except recipe.ConversionError as error: # type: ignore[attr-defined]
|
||||
detail = getattr(error, "body", None) or str(error)
|
||||
raise ValueError(f"DeepSeek-V4.1 prompt conversion failed: {detail}") from error
|
||||
|
||||
return encoding.render_conversation(converted.conversation).prompt
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user