Compare commits

..
Author SHA1 Message Date
minke.yu aa23c29d52 deploy(b300/dynamo): publish default thinking mode from the workers
The AgentX baseline runs with thinking=true/high by default; the Dynamo
frontend only applies a default when the worker publishes
default_thinking_mode. Pass --dyn-default-thinking-mode enabled on both
P and D so the SGLang chat processor matches the sglang baseline.
2026-11-17 22:55:08 +08:00
minke.yu 99603451b9 docs(b300/dynamo): record tokenizer/chat-processor fixes and first-run results
Document --dyn-chat-processor sglang + --dyn-preprocess-workers +
--tokenizer fastokens, the encoding_dsv4 shim, the fastokens propagation
gap in sglang_processor._load_tokenizer, the v41 thinking opt-in fix, and
the warm/cold smoke numbers.
2026-11-17 22:42:19 +08:00
minke.yu 81428f230c deploy(b300/dynamo): restore disaggregation lanes and mount frontend patches
- SGLANG_DISAGGREGATION_QUEUE_SIZE=16 / THREAD_POOL_SIZE=32 (cw baseline
  values; service defaults are 4).
- NCCL all-gather sync-batch env on P; batch spans off until the next
  image build carries the TraceNullContext guard.
- FE mounts the encoding_dsv4 shim and the v41 thinking patch.
2026-11-17 22:29:38 +08:00
minke.yu 2372edd73d observability: skip batch-span links for TraceNullContext requests
TraceNullContext forwards every attribute to itself, so the previous
root_span.get_span_context() probe returned the null context itself and
otel's exporter failed with 'expected bytes, TraceNullContext found'.
Gate on tracing_enable before probing.
2026-11-17 22:27:55 +08:00
minke.yu 34d2224b7f dynamo: add encoding_dsv4 shim for the frontend SGLang chat processor
Dynamo's sglang_post imports upstream sglang.srt.entrypoints.openai.
encoding_dsv4.encode_messages; our fork uses the deepseek-recipe renderer.
Expose the upstream API on top of the recipe (V4.1 spec), returning the
prompt string; tool schemas follow Dynamo's system-message convention.
2026-11-17 22:22:18 +08:00
minke.yu 36f6a9cdb3 fix(b300/dynamo): restore FE_PORT definition 2026-11-17 22:11:31 +08:00
minke.yu 2b75283daa deploy(b300/dynamo): run frontend chat processing on SGLang processor
Use the 1.6.2 frontend tokenizer controls instead of per-worker gateways:
--dyn-chat-processor sglang + --dyn-preprocess-workers N + --tokenizer
fastokens, with explicit deepseekv41 / deepseek-v41 parsers (sglang's
'auto' is not resolved by the frontend parser factory). Also start the
L3 mooncake master from the same image.
2026-11-17 22:10:57 +08:00
minke.yu 0fbd87d739 docs(b300/dynamo): Dynamo does not remove SGLang's bootstrap/prealloc tail
Bootstrap completion is discovered at scheduler pass boundaries via
poll_and_all_reduce_attn_cp_tp_group; Dynamo's frontend still waits on it.
Record the measured prefill_bootstrap vs mooncake-send gap and point at
the sglang-side levers.
2026-11-17 21:51:15 +08:00
minke.yu bcfdd8a1f7 docs(b300/dynamo): spell out Dynamo PD control plane vs SGLang KV data plane
Dynamo's frontend/PrefillRouter owns P/D selection and bootstrap-info
propagation; the SGLang backend still delegates the KV transfer to
sglang's native disaggregation (mooncake + our NVLink patches).
--connector nixl / --kv-transfer-config apply to vLLM/TRT-LLM only.
2026-11-17 21:42:19 +08:00
minke.yu 6f546386ca deploy(b300/dynamo): add Dynamo 1.6.2 image recipe and 4P4D PD startup
- Dockerfile/build script: current B300 baked image (runtime tree identical to
  b300/dynamo) + ai-dynamo/ai-dynamo-runtime 1.6.2 installed with --no-deps.
- start_dynamo_pd.sh: dynamo.frontend :33000 + 4P(dp4)/4D(dp4) workers on
  GPU 0-3/4-7, mooncake NVLink, HiCache L2 + L3 SSD.
- README: compatibility audit (dynamo 1.6.2 unit tests match the nightly
  control run), version matrix, known constraints, open items.
2026-11-17 21:01:52 +08:00
minke.yu 8b67636010 Merge b300/integration into main: 2026-11-17 c580 baseline
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included

This is the base for the next Dynamo layer.
2026-11-17 20:08:08 +08:00
7 changed files with 3347 additions and 2742 deletions
+41
View File
@@ -0,0 +1,41 @@
# Dynamo-enabled B300 serving image.
#
# The sglang runtime tree inside the base image is byte-identical to
# b300/dynamo @ 8b67636010 (the only commits between the baked tree
# 433257c7b0 and 8b67636010 are deploy/docs), so this layer only swaps the
# preinstalled Dynamo 1.6.0.dev snapshot for the released 1.6.2 runtime.
#
# Build:
# docker build -t ymkymx/sglang:dynamo-8b6763601-baked-20261117 .
# or use ./build-dynamo-image.sh (also imports into containerd).
ARG BASE=ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
FROM ${BASE}
ARG DYNAMO_VERSION=1.6.2
ARG PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
LABEL ai.sglang.build.commit="8b67636010e54fc96e5aeba9b0139b38aa18b000"
LABEL ai.dynamo.version="${DYNAMO_VERSION}"
LABEL org.opencontainers.image.description="SGLang B300 (b300/dynamo) + NVIDIA Dynamo ${DYNAMO_VERSION}"
# --no-deps: the base image already carries the full runtime dependency set
# (torch 2.14.1, ai-dynamo deps, nixl 1.5.0). Resolving would try to pull
# sglang==0.5.19 from the `sglang` extra and downgrade torch.
RUN pip uninstall -y ai-dynamo ai-dynamo-runtime >/dev/null 2>&1 || true \
&& pip install --no-cache-dir --no-deps -i "${PIP_INDEX}" \
"ai-dynamo==${DYNAMO_VERSION}" "ai-dynamo-runtime==${DYNAMO_VERSION}"
# CPU-only smoke: both CLIs must import the sglang args we build from source.
RUN python - <<'PY'
import importlib.metadata as m
import sglang
import dynamo.sglang # noqa: F401
import dynamo.frontend # noqa: F401
print("dynamo", m.version("ai-dynamo"), "runtime", m.version("ai-dynamo-runtime"))
print("sglang", sglang.__file__)
PY
RUN python -m dynamo.sglang --help >/dev/null && python -m dynamo.frontend --help >/dev/null
ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
CMD []
+216
View File
@@ -0,0 +1,216 @@
# Dynamo 支持(b300/dynamo)
> 分支:`b300/dynamo`(基点 `8b67636010`,= `main` = `b300/integration` 的 2026-11-17 c580 基线)
> 状态:**代码/镜像/启动脚本已就绪,静态兼容性已验证;GPU 运行时验证待窗口**
> (2026-11-17 时 `ds41-gist-exp` 占满 8 卡)。
本目录是 B300 线切换到 **NVIDIA Dynamo** 的入口:Dynamo frontend + 我们的 sglang
worker(P/D 分离)。它与现有 `ds41-gist-exp` 的区别:**入口 + PD 编排从 g9i gateway
换成 Dynamo frontend(`:33000`)**;**KV 数据面仍是 sglang 原生 disaggregation(mooncake)**。
## 1. 兼容性结论(2026-11-17)
- **不需要 sglang 代码改动**。Dynamo 1.6.2 的 `dynamo.sglang` 后端通过 `sgl.Engine`
公开接口工作,`_compat.py` 对 0.5.18/0.5.19 的兼容层在当前树(main-era)全部命中
新版分支。
- 用 Dynamo 1.6.2 自带单测跑了当前树(`PYTHONPATH=<tree>/python`,镜像内先装
`pytest-asyncio`):**985 passed / 20 failed / 1 skipped**。同一镜像用自带 sglang
nightly 跑对照也是 **985 passed / 20 failed / 1 skipped**,`diff` 后失败用例集合
**逐条相同**(均为视频生成等既有环境相关用例)。→ 无 API / 行为漂移。
- `python -m dynamo.sglang --help`(1398 行,sglang 参数已合并)与
`python -m dynamo.frontend --help` 均 rc=0。
- Dynamo 依赖的 `ServerArgs` 字段(`enable_forward_pass_metrics`、
`forward_pass_metrics_ipc_name`、`elastic_ep_backend`、`dllm_algorithm`、
`incremental_streaming_output`、`kv_events_config`、`allow_auto_truncate` 等)
在当前树全部存在。
复现命令(CPU,容器内):
```bash
docker run --rm --entrypoint bash \
-e PYTHONPATH=/data/ymk/wt-dynamo/python \
-v /data/ymk/wt-dynamo:/data/ymk/wt-dynamo:ro \
<dynamo-image> -c 'python -m dynamo.sglang --help >/dev/null && echo ok'
```
## 2. 版本矩阵
| 组件 | 版本 | 备注 |
|---|---|---|
| sglang | `b300/dynamo` @ `8b67636010` | 与镜像内已烘源码 `433257c7b0` 的 `python/sglang` **逐字节一致**(中间只有 deploy/docs 提交) |
| Dynamo | **1.6.2**(`ai-dynamo` + `ai-dynamo-runtime`) | 基底镜像原本带 `1.6.0.dev20261019`,Dockerfile 覆盖 |
| nixl | 1.5.0(`nixl-cu13` 1.5.0) | 基底已带;Dynamo 的 SGLang 后端**不用**它做 KV 传输 |
| torch | 2.14.1+cu130 | 基底已带,不因 Dynamo 变化 |
| sglang-kernel | 0.4.10 | 基底已带,与当前树 pin 一致 |
| mooncake | 0.3.14(L3 SSD) | 基底已带(见 `../README.md`) |
- KV 传输仍是 **sglang 原生 disaggregation + mooncake**;`--connector nixl` 只对
vLLM/TRT-LLM 生效,对 SGLang 是 no-op(10-26 已确认,本脚本不带它)。
- NVLink:`MC_INTRANODE_NVLINK=true` + `MC_INTRA_NVLINK=true` + `--pid host` +
宿主已烘的 NVLink host 修复。
### 2.1 PD 的两层要分开看(常被混淆)
| 层 | 由谁实现 | 在本部署里的表现 |
|---|---|---|
| **PD 编排 / 路由**:选 P、选 D、把 `bootstrap_info` 从 P 传给 D、流式回传 | **Dynamo 自己的 Rust runtime + frontend PrefillRouter**(不是 sglang-router/mini_lb) | `dynamo.frontend :33000`;Dynamo 的 `sglang` handler 只负责把 bootstrap 信息塞进/取出 `Engine.async_generate` |
| **KV 数据面**:bootstrap 握手后的 KV 传输 | **sglang 原生 disaggregation**(`--disaggregation-mode` + `--disaggregation-transfer-backend mooncake`) | P/D 日志的 `Using Intra-Node NVLink transport`;10-26 的 NVLink host 补丁仍然生效 |
Dynamo 的 `--connector nixl` / `--kv-transfer-config` 只对 vLLM/TRT-LLM 生效(vLLM 需要
`--kv-transfer-config '{"kv_connector":"NixlConnector",...}'`);SGLang 后端把
`bootstrap_host/port/room` 直接传给 `Engine.async_generate`,传输由 sglang 的 bootstrap
server + mooncake 完成(`prefill_handler.py` 产出 `disaggregated_params`,
`decode_handler.py` 消费 `bootstrap_info`)。所以 P/D 两侧仍然必须写
`--disaggregation-mode`,且 `--disaggregation-transfer-backend` 要与 sglang 侧一致。
> 结论:**编排是 Dynamo 的,KV 线是 sglang 的**。两者不是同一套,但也不是各走各的——
> Dynamo 控制面负责“谁和谁配对”,sglang 数据面负责真正搬 KV。
### 2.3 Dynamo 1.6.2 的 tokenizer / chat processor(实测)
DeepSeek-V4.1 没有 HF `chat_template`,默认的 Rust processor 无法渲染;需要用
SGLang 的 pre/post processor:
```
--dyn-chat-processor sglang \
--dyn-preprocess-workers 8 \
--tokenizer fastokens --trust-remote-code \
--tool-call-parser deepseekv41 \
--reasoning-parser deepseek-v41
```
本次踩通/修掉的四个点:
1. **缺 `enencoding_dsv4`**:`--dyn-chat-processor sglang` 会 import
`sglang.srt.entrypoints.openai.enencoding_dsv4`;我们的 fork 用 deepseek-recipe,没有该模块,
chat 直接 500。已在 `b300/dynamo` 加 shim `enencoding_dsv4.py`(基于 recipe 的
`render_conversation`),token ids 与生产 `DeepSeekRecipe.encode` 逐 token 相同。
2. **`--tokenizer fastokens` 不会自动传给 SGLang processor**:该 flag 只作用于 Dynamo 的
Rust model-card tokenizer;`sglang_processor._load_tokenizer()` 调 `get_tokenizer()`
时默认 `huggingface`。需要把 `FrontendConfig.tokenizer_backend` 传进去(FE hostPath 补丁),
日志出现 `fastokens backend enabled - transformers patched successfully` 才算生效。
3. **v41 thinking 解析**:Dynamo 的 `_THINKING_OPT_IN` 缺 `deepseek-v41`,显式
`chat_template_kwargs.thinking=true` 时 `reasoning_content` 不会拆出来。已补上;
默认请求仍是 chat(worker 没有发布 `default_thinking_mode`;生产默认 thinking 需要
register 侧补丁,TODO)。
4. **`--gateway-workers N`(worker 侧)** 是另一条路:给默认 Rust processor 用,启动 N 个
tokenizer/gateway 进程并设置 SGLang `--tokenizer-worker-num`;`--use-sglang-tokenizer`
已 deprecated。本次走 frontend processor,不需要它。
另:`encoding_dsv4`/`thinking` 的 FE 补丁目前以 hostPath 挂载在容器里,尚未烘进镜像;
重建镜像时要一起带上(见 `start_dynamo_pd.sh` 的 `-v` 列表)。
### 2.4 2026-11-17 首次运行结果(4P(dp4)+4D(dp4))
- 启动约 7 分钟;P `max_total_num_tokens=16,157,696`、D `58,859,520`(与 11-17
`ds41-gist-exp` 基线一致);P/D 均出现 `Using Intra-Node NVLink transport`。
- chat / tool_calls / reasoning_content 均可;prefix cache 正常(16k prompt 第二轮
cached 15,872;80k 第二轮 cached 79,872)。
- 32 并发 × ~900 token prompt(85% 命中,warm):**wall 0.62 s,32/32 成功**;
16k 冷 0.47 s / 热 0.24 s;80k 热 0.25 s(冷 80k ~1.04 s,HF tokenizer 时的数字)。
**重启后前 1–2 波会明显慢(8–9 s)是 warmup 伪影,benchmark 必须先热身。**
- P 的 OTel batch-span link 会报 `expected bytes, TraceNullContext found`(旧镜像);
已在分支修 `batch_telemetry.py`,本次运行用 `SGLANG_BATCH_SPAN=0` 规避。
### 2.2 Dynamo 不会消除 sglang 的 bootstrap / 握手尾巴
Dynamo 换掉的是控制面,sglang 内部的 bootstrap 路径原样保留:
- P 侧 `PrefillBootstrapQueue.pop_bootstrapped()` 与 `resolve_waiting_queue_bootstrap()`
仍然在 scheduler 每个 pass 里做 `poll_and_all_reduce_attn_cp_tp_group(...)`
(`disaggregation/prefill.py:495-556`、`638-675`);bootstrap 完成只在 pass 边界被发现。
- D 侧 `DecodePreallocQueue` / `_ensure_prefill_info` / 传输接收队列也一样。
- 因此 `prefill_bootstrap`(P queue entry → bootstrap done)与 `decode_transferred`
(等 P 完成 + 收 KV)的口径不变,Dynamo frontend 必须等它们完成才会开始 decode;
长 pass / DP skew / 大 batch 会按 pass 放大这个延迟。
实测参考(当前栈):c620 下 `prefill_bootstrap` p50 0.23–0.36 s、p95 1.66 s(fresh)/ 5.05 s
(不重启),而 `mooncake_worker_send_session` p50 只有 2.6 ms —— 卡的是握手/轮询,不是搬 KV。
Dynamo 的收益在入口/路由/可观测性;要压 bootstrap 尾部必须改 sglang scheduler 的轮询/预分配路径
(或先用 optimistic prefill + `hicache-write-policy write_back` 把 prefill 与握手重叠)。
## 3. 构建镜像
```bash
cd deploy/b300-k8s/dynamo
./build-dynamo-image.sh # 构建 + ctr import
# IMPORT=0 只构建;TAG=... 自定义 tag
```
默认产物:`ymkymx/sglang:dynamo-8b6763601-baked-20261117`
(= `sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:...`,同一 image ID)。
要点:Dynamo 必须 `pip install --no-deps`。基底已带全部运行时依赖;一旦让 pip 解析,
`ai-dynamo` 的 `sglang` extra 会试图装 `sglang==0.5.19` 并降级 torch。
## 4. 启动(docker,4P(dp4)+4D(dp4))
先停掉占卡的工作负载并确认 GPU 0-7 为 0 MiB,再:
```bash
./start_dynamo_pd.sh
./smoke.sh # /v1/models + /v1/chat/completions
tail -f /data/ymk/gist-dsv41/dyn-{fe,p,d}.log
```
端口:frontend **33000**、P **30000**、D **30300**;P/D 的 sglang 指标仍在原端口,
Dynamo 自己的指标在 frontend。
> ⚠️ 启动耗时与普通 P/D 栈相同(P/D 各 ~6-10 分钟;`/root/.cache/sglang` 挂载可复用
> autotune/JIT 缓存)。**先等 P/D ready 再开压测**:Dynamo 有冷启动首波长尾
> (空闲后首波 3–22s,暖机后消失)。
## 5. 与 2026-10-26 Dynamo 试验的差异
| 项 | 10-26 试验 | 本目录 |
|---|---|---|
| Dynamo | runtime 镜像 `sglang-runtime:1.6.0` | PyPI **1.6.2**(烘进 B300 镜像) |
| 模型 | `DeepSeek-V4.1-Flash` | `DeepSeek-V4.1-Flash-1017`(served name 保持原路径) |
| 并行 | tp4 + tp4(`--moe-a2a-backend none`) | **dp4 + dp4**(dp-attention + dp-lm-head + megamoe + flashmla_mega) |
| 入口 | `dynamo.frontend :33000` | 同上 |
| 缓存 | L2/L3 32GB | 当前生产 L3 SSD 4TB + `write_through` |
| 门控 | `SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0` | 同(依赖树内 `7f6fe22445`/`ee9170e7bb` fail-soft) |
dp4 是 DSpark PD 的硬要求:两侧 `attn_tp_size` 必须相等。dp4 下 attn_tp = tp/dp = 1,
P/D 一致;若一侧 tp4、另一侧 dp4,`disaggregation/common/conn.py:1355` 会直接拒绝
(`DSpark PD requires the same TP size on both servers`)。
## 6. 已知约束 / 坑
1. **`--host 0.0.0.0` 必须**:sglang bootstrap server 跟随 `--host`;写 127.0.0.1 时
D 侧拿不到 prefill parallel info → `HTTP 500: Decode handshake failed`。
2. **`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python`**:baked 镜像没有
`rust_tree_core` `.so`(fake-cargo 阻止现场编译);1M 上下文下 unified cache 会选
Rust TreeCore 并 `ModuleNotFoundError`。本脚本已设为 `python`。
3. **Dynamo 1.6.2 安装用 `--no-deps`**:见 §3。`aisimulate` 在清华源只有
`0.13.0.dev2`(Dynamo 1.6.2 元数据要 `0.13.0`),实测 import 正常。
4. **`--dyn-preprocess-workers`** 取代了旧报告的 “gateway workers”:1.6.2 默认
auto(每个 model 最多 2),可显式设 4–8 提升高并发 tokenize 吞吐;仅在
`--dyn-chat-processor sglang` 下生效。
5. **Dynamo frontend 没有 g9i 的准入闸门**(`--g9i-p-queue-reqs-cap` 等)。Dynamo 自己的
flow control(`--router-queue-threshold` / `--load-aware`)在 10-26 实测无效,
有效闸门仍是客户端/网关并发上限。
6. **P/D 的 health / readiness**:`dynamo.sglang` worker 通过 discovery 注册,不使用
sglang 的 `/health` 作为就绪判据;k8s 化时探针应指向 frontend,或改用 dynamo 的
health payload(`dynamo.sglang.health_check`)。
7. `--enable-trace` 的 OTLP 服务名分别为 `dyn-p` / `dyn-d`(与旧试验一致);
Jaeger `:4317` 可用。
8. 端口分配遵守 `30000-32767` 固定端口纪律;frontend 33000 与现有栈不冲突。
## 7. 待办
- [ ] **GPU 冒烟**:等 `ds41-gist-exp` 停/窗口;验证 PD 全链路、NVLink transport 日志
(`Using Intra-Node NVLink transport`)、DSpark accept len、L2/L3 命中。
- [ ] **性能基线**:c580/1200s(同 `baselines/20261117-dtok16-fastokens` 口径)对比
g9i gateway 栈;关注 Dynamo frontend 的 tokenize 开销与冷启动长尾。
- [ ] **k8s cw manifest**:`ds41-dynamo`(frontend + p + d + mooncake-master),
探针/服务注册按 Dynamo discovery 设计;本目录先提供 docker 形态。
- [ ] 视需要把 namespace / Chat-Completions 工具展平(`b300/feat-recipe-ns` 已进
`b300/integration`)在 Dynamo frontend 侧的行为纳入 `preflight/issue-set`。
## 8. 参考
- 旧运行记录:`D:\B300\experiments\dsv41-gist-deepinfer\REPORT-DYNAMO-20261026.md`
- 启动脚本(旧,tp4):`D:\B300\experiments\dsv41-gist-deepinfer\start_dynamo_pd.sh`
- Dynamo 仓库(本地):`D:\B300\dynamo`(1.6.0 源码树,仅作 API 参考;实际运行用 1.6.2 wheel)
- L3 SSD 基线:`deploy/b300-k8s/README.md`
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Build (and optionally load into containerd) the Dynamo-enabled B300 image.
#
# ./build-dynamo-image.sh # build + ctr import
# IMPORT=0 ./build-dynamo-image.sh # build only
# TAG=my-tag ./build-dynamo-image.sh # custom tag
set -euo pipefail
cd "$(dirname "$0")"
BASE=${BASE:-ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115}
TAG=${TAG:-dynamo-8b6763601-baked-20261117}
REPO=${REPO:-sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang}
IMPORT=${IMPORT:-1}
DYNAMO_VERSION=${DYNAMO_VERSION:-1.6.2}
echo "base=$BASE tag=$TAG dynamo=$DYNAMO_VERSION"
docker build \
--build-arg BASE="$BASE" \
--build-arg DYNAMO_VERSION="$DYNAMO_VERSION" \
-t "ymkymx/sglang:$TAG" \
-f Dockerfile .
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
docker run --rm --entrypoint bash "ymkymx/sglang:$TAG" -c '
python - <<PY
import importlib.metadata as m, importlib.util
import dynamo.sglang, dynamo.frontend # noqa: F401
print("ok: ai-dynamo", m.version("ai-dynamo"), "| sglang", importlib.util.find_spec("sglang").origin)
PY
python -m dynamo.sglang --help >/dev/null
python -m dynamo.frontend --help >/dev/null
'
if [ "$IMPORT" = "1" ]; then
echo "importing $REPO:$TAG into containerd (k8s.io) ..."
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
fi
echo "done: $REPO:$TAG"
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# Dynamo frontend smoke test.
# ./smoke.sh [base_url] [model_name]
set -euo pipefail
BASE=${1:-http://127.0.0.1:33000}
MODEL=${2:-/data/models/DeepSeek-V4.1-Flash}
echo "== /v1/models"
curl -fsS "$BASE/v1/models" | head -c 600; echo
echo "== chat"
curl -fsS "$BASE/v1/chat/completions" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"2+3=? Answer with the number only.\"}],\"max_tokens\":64,\"temperature\":0}" \
| head -c 1200; echo
+183
View File
@@ -0,0 +1,183 @@
#!/usr/bin/env bash
# 4P(dp4)+4D(dp4) PD on NVIDIA Dynamo 1.6.2 + our B300 sglang (b300/dynamo @ 8b67636010).
#
# frontend :33000 (Dynamo router; replaces the g9i gateway)
# prefill :30000 GPU 0-3, mooncake NVLink, HiCache L2 + L3 SSD
# decode :30300 GPU 4-7
#
# Differences vs the 2026-10-26 Dynamo bring-up (REPORT-DYNAMO-20261026.md):
# * dp4 (tp4 + dp-attention + dp-lm-head) on both sides instead of tp4, matching
# the current production stack. Both sides must expose attn_tp_size==1 or the
# DSpark PD handshake rejects the pair ("same TP size on both servers").
# * image carries Dynamo 1.6.2 (the 10-26 run used the 1.6.0 runtime image).
# * L3 SSD mooncake 0.3.14 + the 1017 model / fastokens tokenizer.
#
# NOT validated on GPU yet (2026-11-17: ds41-gist-exp owns all 8 GPUs).
set -uo pipefail
IMG=${IMG:-ymkymx/sglang:dynamo-8b6763601-baked-20261117}
MODEL=${MODEL:-/data/models/DeepSeek-V4.1-Flash-1017}
SERVED=${SERVED:-/data/models/DeepSeek-V4.1-Flash}
D=${D:-/data/ymk/gist-dsv41}
STORE=${STORE:-$D/dynamo-store}
L3_CONF=${L3_CONF:-/data/ymk/ds41/mooncake-store-l3ssd.json}
JITCACHE=${JITCACHE:-/data/ymk/cache/sglang}
FE_PORT=${FE_PORT:-33000}
# Frontend tokenizer/chat processing (1.6.2):
# --dyn-chat-processor sglang frontend does SGLang-native template rendering,
# tokenization, tool/reasoning parsing
# --dyn-preprocess-workers N ProcessPool of N tokenizer/template workers
# --tokenizer fastokens fastokens BPE encoder (falls back to HF)
# Parser names must be explicit: sglang's "auto" is resolved at server startup
# and is not understood by the frontend's parser factory.
PREPROCESS_WORKERS=${PREPROCESS_WORKERS:-8}
TOOL_PARSER=${TOOL_PARSER:-deepseekv41}
REASONING_PARSER=${REASONING_PARSER:-deepseek-v41}
FE_ARGS=${FE_ARGS:---router-mode round-robin \
--dyn-chat-processor sglang \
--dyn-preprocess-workers $PREPROCESS_WORKERS \
--tokenizer fastokens --trust-remote-code \
--tool-call-parser $TOOL_PARSER \
--reasoning-parser $REASONING_PARSER}
mkdir -p "$STORE"; chmod 777 "$STORE" 2>/dev/null || true
docker rm -f dyn-fe dyn-p dyn-d >/dev/null 2>&1
sleep 3
# Dynamo/sglang children can outlive the container when --pid host is used.
for p in $(ps -eo pid,comm | awk '$2 ~ /^sglang::/ {print $1}'); do kill -9 "$p" 2>/dev/null; done
sleep 2
# L3 store master (not part of the Dynamo graph). Same pod/port as the cw master.
if ! docker ps --format '{{.Names}}' | grep -qx dyn-mooncake; then
docker rm -f dyn-mooncake >/dev/null 2>&1
docker run -d --name dyn-mooncake --network host "$IMG" \
/opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 \
--eviction_high_watermark_ratio=0.90 \
--enable_offload=true --offload_on_evict=false \
--promotion_on_hit=false --offload_force_evict=false \
> "$D/dyn-mooncake.log" 2>&1
sleep 3
fi
COMMON_ENVS=(
-e SGLANG_RAGGED_VERIFY_MODE=static
-e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1
-e SGLANG_DSV4_KV_LAYOUT=v41_new
-e MC_INTRANODE_NVLINK=true
-e MC_INTRA_NVLINK=true
-e SGLANG_MOONCAKE_SEND_AUX_TCP=true
# The baked image has no rust_tree_core .so; force the Python radix tree.
-e SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
-e SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600
-e SGLANG_DISAGGREGATION_QUEUE_SIZE=16
-e SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=32
-e SGLANG_BATCH_TELEMETRY=1
)
P_ENVS=(
-e SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1
-e SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1
-e SGLANG_OPT_DSV41_BF16_INDEX_LOGITS=1
-e SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536
-e SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0
-e SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=$L3_CONF
-e SGLANG_HICACHE_PREFETCH_IO_THREADS=4
-e SGLANG_HICACHE_BACKUP_IO_THREADS=4
-e SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1
-e SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH=1
# The baked image predates the TraceNullContext guard; keep batch spans off
# until the next image build (histograms are unaffected).
-e SGLANG_BATCH_SPAN=0
)
D_ENVS=(
-e SGLANG_ENABLE_METRICS_DP_ATTENTION=1
)
# ---------------------------------------------------------------- frontend
docker run -d --name dyn-fe --network host --entrypoint bash \
-v /data:/data -v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/wt-dynamo/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:ro \
-v /data/ymk/dynamo-patch-sglang_prepost.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_prepost.py:ro \
-v /data/ymk/dynamo-sglang_processor.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_processor.py:ro \
"$IMG" \
-c "python -m dynamo.frontend --discovery-backend file --http-port $FE_PORT $FE_ARGS \
> $D/dyn-fe.log 2>&1"
# ---------------------------------------------------------------- prefill
docker run -d --name dyn-p --gpus all -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
"${COMMON_ENVS[@]}" "${P_ENVS[@]}" \
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
-v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
"$IMG" python -m dynamo.sglang \
--endpoint dyn://dynamo.prefill.generate --discovery-backend file \
--disaggregation-mode prefill \
--trust-remote-code --tokenizer-backend fastokens \
--model-path "$MODEL" --served-model-name "$SERVED" \
--reasoning-parser auto --tool-call-parser auto \
--dyn-default-thinking-mode enabled \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-dp-attention-local-control-broadcast \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-p \
--watchdog-timeout 3600 \
> "$D/dyn-p.log" 2>&1
# ---------------------------------------------------------------- decode
docker run -d --name dyn-d --gpus all -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
"${COMMON_ENVS[@]}" "${D_ENVS[@]}" \
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
-v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
"$IMG" python -m dynamo.sglang \
--endpoint dyn://dynamo.backend.generate --discovery-backend file \
--disaggregation-mode decode \
--trust-remote-code --tokenizer-backend fastokens \
--model-path "$MODEL" --served-model-name "$SERVED" \
--reasoning-parser auto --tool-call-parser auto \
--dyn-default-thinking-mode enabled \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-d \
--watchdog-timeout 3600 \
> "$D/dyn-d.log" 2>&1
sleep 12
docker ps --format '{{.Names}}\t{{.Status}}' | grep -E '^dyn-' || true
echo "---- frontend log"; tail -3 "$D/dyn-fe.log" 2>/dev/null
echo "---- prefill log"; tail -3 "$D/dyn-p.log" 2>/dev/null
echo "---- decode log"; tail -3 "$D/dyn-d.log" 2>/dev/null
echo "[$(date +%H:%M:%S)] launched: FE=$FE_PORT P=30000 D=30300 (image $IMG)"
@@ -0,0 +1,105 @@
# SPDX-License-Identifier: Apache-2.0
"""Compatibility shim for Dynamo's SGLang chat processor.
Upstream SGLang builds expose ``sglang.srt.entrypoints.openai.encoding_dsv4``
with ``encode_messages(messages, *, thinking_mode, reasoning_effort) -> str``.
Dynamo's frontend (``dynamo.frontend.sglang_prepost``) imports that module to
render DeepSeek-V4/V4.1 prompts when the model ships no HF chat template.
Our DeepSeek-V4 fork replaced the encoder with the ``deepseek-recipe``
renderer (``sglang.srt.parser.deepseek_recipe_renderer``). This module exposes
the upstream API on top of the recipe, so ``--dyn-chat-processor sglang``
works without installing a second SGLang build.
The returned string is tokenized by the caller. For the V4.1 checkpoint the
resulting ids match ``DeepSeekRecipe.encode()`` (the native serving path)
byte-for-byte; the renderer is the same code path, only the tokenizer call
moves to the caller.
"""
from __future__ import annotations
import copy
import json
from functools import lru_cache
from typing import Any, Optional
from sglang.srt.parser.deepseek_recipe_renderer import (
IMAGE_TOKEN,
MODEL_NAMES,
load_deepseek_recipe,
pre_normalize,
)
# The Dynamo frontend uses this encoder for the V4 family. Our deployment is
# DeepSeek-V4.1-Flash; V4-Pro checkpoints need the ``dsv4`` spec instead.
_CHAT_ENCODING_SPEC = "dsv41"
@lru_cache(maxsize=1)
def _renderer():
recipe = load_deepseek_recipe()
special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
encoding = recipe.DeepseekV41Encoding().with_special_tokens(special_tokens)
options = recipe.ConversionOptions().with_special_tokens(special_tokens)
return recipe, special_tokens, encoding, options
def encode_messages(
messages: list[dict[str, Any]],
*,
thinking_mode: str = "chat",
reasoning_effort: Optional[str] = None,
) -> str:
"""Render OpenAI-style ``messages`` to a DeepSeek-V4.1 prompt string.
Mirrors upstream ``encoding_dsv4.encode_messages``:
* ``thinking_mode`` is ``"thinking"`` or ``"chat"``;
* ``reasoning_effort`` is a named tier (``"high"`` / ``"max"``) or None;
* tool schemas are attached to the first system message by the caller
(Dynamo's convention), not passed as a separate argument.
Raises ``ValueError`` for prompt shapes the recipe rejects.
"""
recipe, _special_tokens, encoding, options = _renderer()
encoding_messages = copy.deepcopy(messages)
tools = None
if encoding_messages:
tools = encoding_messages[0].pop("tools", None)
if not tools:
tools = None
# Dynamo already flattens message content and re-serializes tool-call
# arguments; keep this defensive normalization for direct callers.
for message in encoding_messages:
if message.get("content") is None:
message["content"] = ""
for call in message.get("tool_calls") or []:
function = call.get("function") if isinstance(call, dict) else None
if isinstance(function, dict) and isinstance(
function.get("arguments"), (dict, list)
):
function["arguments"] = json.dumps(
function["arguments"], ensure_ascii=False
)
body: dict[str, Any] = {
"messages": encoding_messages,
"thinking": thinking_mode == "thinking",
}
if tools:
body["tools"] = tools
if reasoning_effort is not None:
body["reasoning_effort"] = reasoning_effort
normalized = pre_normalize(body)
normalized["model"] = MODEL_NAMES[_CHAT_ENCODING_SPEC]
try:
converted = recipe.ChatCompletionRequest(normalized).convert(options)
except recipe.ConversionError as error: # type: ignore[attr-defined]
detail = getattr(error, "body", None) or str(error)
raise ValueError(f"DeepSeek-V4.1 prompt conversion failed: {detail}") from error
return encoding.render_conversation(converted.conversation).prompt
File diff suppressed because it is too large Load Diff