[Doc] [AMD] cookbook: Refresh DeepSeek V4 AgentX PD recipe optimizations (#44454)

This commit is contained in:
Duyi-Wang
2026-10-16 23:34:01 +08:00
committed by GitHub
parent 2c2986be2b
commit ed2ebad342
2 changed files with 76 additions and 13 deletions
@@ -948,13 +948,13 @@ docker run \
python3 -m sglang.launch_server <role args below>
```
The Deployment and Playground Docker outputs use this `v0.5.21` build for the MI355X Pro and Pro Official FP4 multi-node recipes. It is the build the PD roles ran on, and the first one carrying the Aiter MegaMoEv2 kernels the high-throughput prefill role needs. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.
The Deployment and Playground Docker outputs retain this `v0.5.21` base image for the MI355X Pro and Pro Official FP4 multi-node recipes. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.
For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds.
#### Role-based configuration
Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed.
Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. The Pro Official commands additionally apply the kernel and routing updates described below.
| Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size |
|---|---|---|---|---|
@@ -966,7 +966,7 @@ These are two separate workers: neither receives `--nnodes 2` or a cross-node TP
`--cuda-graph-bs-decode` covers 1 to `--max-running-requests / attn_dp_size`: the full value on the TP-only rows and an eighth on the DP8 rows. Each strategy sets both workers' admission limit and decode's graph sizes together. The TP-only points use `--mem-fraction-static 0.86` and SWA ratio `0.1`; the DP8 points use `0.92` and `0.15`.
All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. Both high-throughput workers enable the FP8 unified KV layout. The commands below show concurrency 256.
All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. All Pro Official PD workers enable `SGLANG_DSV4_UNIFIED_KV_FP8=1`. Low-Latency and Balanced also set `SGLANG_DSV4_DENSE_MXSCALE=1` on both roles; High-Throughput sets mxscale on decode only. The decode dispatch budget is `64 × (DSpark block size + 1)`: `MORI_MAX_DISPATCH_TOKENS_DECODE=448` at concurrency 16 and `256` at concurrency 48 or 256. The commands below show concurrency 256.
Select **Pro** to use the original checkpoint's **EAGLE/MTP 3-1-4** head with the same topology and admission limits. Its commands contain no DSpark parameters. In the Playground, **Off (greedy)** removes speculative decoding. Enabling **HiCache** automatically turns off **Unified Cache External Linker** and removes UMBP's socket configuration and optimistic prefill. Enabling the linker turns HiCache off.
@@ -1047,6 +1047,7 @@ python3 -m sglang.launch_server \
```bash Command
GPU_MAX_HW_QUEUES=5 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_DSV4_DENSE_MXSCALE=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
@@ -1102,11 +1103,29 @@ On the prefill env:
These recipes leave `MORI_SHMEM_MODE` unset and use MORI's default `STATIC_HEAP`. The pinned Aiter MegaMoEV2 implementation allocates its buffers through `mori_shmem_create_tensor()` / `shmem_malloc()` after SGLang initializes MORI shmem. This requires the symmetric heap; the separate MORI CCO allocator is not used by this implementation. If your launch script exports `MORI_SHMEM_MODE=ISOLATION`, unset it before launching MegaMoE: its peer-pointer lookup requires the heap mapping. Keep `MORI_SHMEM_HEAP_SIZE=8G` for the prefill recipe above.
</Note>
**Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1`**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way.
**Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1` at every Pro Official PD concurrency**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way.
#### Router configuration
Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. Decode holds no reusable prefix, so `--decode-policy round_robin` spreads it evenly. `--dp-aware` takes effect only when the workers run DP attention.
Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. The TP-only presets keep `--decode-policy round_robin`. Pro Official's DP-attention presets (concurrency 192–512, including the High-Throughput preset at 256) instead use:
```bash Router settings
SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests \
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill http://<prefill-host>:30000 \
--decode http://<decode-host>:30100 \
--host 0.0.0.0 --port 8000 \
--policy consistent_hashing --dp-aware \
--decode-policy power_of_two \
--worker-startup-check-interval 1 \
--cache-threshold 0.3 \
--balance-abs-threshold 2 --balance-rel-threshold 1.1 \
--disable-circuit-breaker --health-failure-threshold 100 \
--health-check-timeout-secs 600 --health-check-interval-secs 30
```
Power-of-two compares in-flight request counts across decode DP ranks. The one-second load polling interval replaces the 30-second default. Check for `Power-of-two load metric: LeastRequests` in the router startup log.
The multi-node router presets retain the default request timeout. The standalone high-throughput Playground preset uses `--request-timeout-secs 14400`. At high concurrency the prefill role queues many requests. When the default 1800 s passes the router aborts the request, and an abort that lands while MORI is writing KV cache over RDMA leaves the transfer neither finished nor failed: that prefill worker can no longer send KV cache to decode, and you have to restart it.
@@ -1114,6 +1133,34 @@ The multi-node router presets retain the default request timeout. The standalone
`--request-timeout-secs` is the timeout of the HTTP request from the router to the worker, not a limit on queue time. At 14400 the router also waits four hours before giving up on a worker that has stopped responding. That is acceptable where prefill queues are long, which is why the lower-concurrency recipes keep the default. Lower it again if you lower the concurrency.
</Warning>
#### Additional Pro Official concurrency points
The panel keeps its c16 / c48 / c256 strategy presets. Apply the following overrides to the **Multi-Nodes** commands to reproduce the optimized recipe's endpoints. Here `cN` means a target concurrency of N.
**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 2` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Benchmark with concurrency 1.
**c32:** start from Low-Latency, keep TP4 → TP8 and block size 6, set both admission limits to 64, and capture decode graph sizes 1–64. Both roles use FP8 KV and mxscale, as at c16 and c48.
**c192 / c384 / c512:** start from High-Throughput. Keep TP8+DP8+EP8 prefill → TP8+DP8 decode, DSpark block size 3 (4 draft tokens), and decode dispatch budget 256. Set both admission limits to 384 / 768 / 1024 and capture decode graph sizes 1–48 / 1–96 / 1–128, respectively. Keep FP8 KV on both roles, mxscale on decode only, and the power-of-two router settings above. At c384 and c512, add `--request-timeout-secs 14400` to the router.
**c512 prefill EPLB:** additionally set these environment variables on the prefill worker:
```bash Prefill environment
export SGLANG_AITER_MEGA_RANK_SYNC=1
export SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1
export SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1
```
Append these flags to its launch command:
```text Prefill flags
--enable-eplb
--eplb-rebalance-num-iterations 200
--expert-distribution-recorder-mode stat
```
Keep shared-expert fusion enabled. Rank synchronization lets idle DP ranks participate in MegaMoE dispatch; the prefill-only mode records load and advances rebalancing on synchronized extend passes. Do not set `--expert-balancedness-report-mode` with this mode: it can deadlock prefill on the first request. Decode uses TP MoE and receives none of these EPLB settings. EPLB is specific to c512; the c256 panel preset leaves it disabled.
### 3.8 Unified Cache External Linker (UMBP)
The **unified cache external linker** is the second way to move KV cache off the GPU. The linker is the SGLang feature, and you enable it with `--enable-unified-cache-external-linker`. UMBP is one store the linker can use; you select it with `--unified-cache-external-linker-backend mori`. The Playground card uses the feature name, not the store name.
@@ -52,6 +52,8 @@ export const config = {
// AgentX 1P1D recipe: InferenceX PR #3664, commit df5e433239d7.
// The strategy presets use c16 / c48 / c256. Pro uses its MTP head;
// Pro Official is the InferenceX checkpoint. Acceptance simulation is omitted.
// Pro Official kernel/routing updates follow the optimized AgentX recipe;
// see section 3.7 for the c1 / c512 overrides.
overlayDims: [
{ id: "pdMode", title: "PD role", default: "prefill",
showWhen: (s) => s.hw === "mi355x" && s.quant === "fp4"
@@ -73,13 +75,15 @@ export const config = {
const tp = dp || !prefill ? 8 : 4;
const image = "lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001";
const router = [
...(official && dp ? ["env SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests"] : []),
"python3 -m sglang_router.launch_router",
" --pd-disaggregation",
" --prefill http://<prefill-host>:30000",
" --decode http://<decode-host>:30100",
" --host 0.0.0.0 --port 8000",
" --policy consistent_hashing --dp-aware",
" --decode-policy round_robin",
` --decode-policy ${official && dp ? "power_of_two" : "round_robin"}`,
...(official && dp ? [" --worker-startup-check-interval 1"] : []),
" --cache-threshold 0.3",
" --balance-abs-threshold 2 --balance-rel-threshold 1.1",
" --disable-circuit-breaker --health-failure-threshold 100",
@@ -111,8 +115,12 @@ export const config = {
// SGE follows NIC capabilities; MORI already selects 2 on AMD AINIC.
`GPU_MAX_HW_QUEUES=${dp ? 5 : 2}`,
...(prefill ? ["UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock"] : []),
// Both PD peers must use the same KV layout. TP prefill and all
// decode points use mxscale; MegaMoE prefill keeps its existing GEMMs.
...(official || dp ? ["SGLANG_DSV4_UNIFIED_KV_FP8=1"] : []),
...(official && (!dp || !prefill) ? ["SGLANG_DSV4_DENSE_MXSCALE=1"] : []),
...(official && !prefill ? [`MORI_MAX_DISPATCH_TOKENS_DECODE=${64 * (gamma + 1)}`] : []),
...(dp ? [
"SGLANG_DSV4_UNIFIED_KV_FP8=1",
// Shared-expert/gatherv default off; reduce-scatter defaults on in ROCm.
...(prefill ? ["SGLANG_DP_USE_REDUCE_SCATTER=0"] : [
"SGLANG_SHARED_EXPERT_TP1=1",
@@ -679,6 +687,7 @@ sgl-eval run mmmu_pro \\
{ mode: "prefill",
when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["low-latency"] },
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1"],
stripFlags: ["--prefill-decode-interval"],
flags: [
"--load-balance-method round_robin",
@@ -697,6 +706,8 @@ sgl-eval run mmmu_pro \\
{ mode: "decode",
when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["low-latency"] },
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1",
"MORI_MAX_DISPATCH_TOKENS_DECODE=448"],
stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"],
flags: [
"--load-balance-method round_robin",
@@ -716,6 +727,7 @@ sgl-eval run mmmu_pro \\
{ mode: "prefill",
when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["balanced"] },
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1"],
stripFlags: ["--prefill-decode-interval"],
flags: [
"--tp 4",
@@ -734,6 +746,8 @@ sgl-eval run mmmu_pro \\
{ mode: "decode",
when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["balanced"] },
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1",
"MORI_MAX_DISPATCH_TOKENS_DECODE=256"],
stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"],
flags: [
"--load-balance-method round_robin",
@@ -835,7 +849,8 @@ sgl-eval run mmmu_pro \\
when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["high-throughput"] },
// Same value as prefill — the handshake compares KV layouts.
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1"],
env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1",
"MORI_MAX_DISPATCH_TOKENS_DECODE=256"],
stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"],
flags: [
"--load-balance-method round_robin",
@@ -876,9 +891,9 @@ sgl-eval run mmmu_pro \\
// MI355X fronts the 1P x 1D agentic pair with a cache-aware router rather
// than the default round-robin: consistent_hashing keeps a conversation on
// the prefill worker that already holds its prefix (decode holds no
// reusable prefix, so it stays round-robin), and the tight balance
// thresholds stop that affinity from starving the peer at the small
// running-request ceiling the roles above use. Health checking stays
// reusable prefix, so Pro Official DP uses power-of-two; other presets
// keep round-robin). The tight balance thresholds stop that affinity
// from starving the peer at the small running-request ceiling above. Health checking stays
// enabled here (unlike the default's 999999s interval) because a long
// agentic run should notice a wedged worker, but it is slack enough that
// a multi-minute prefill is not mistaken for a failure.
@@ -896,13 +911,14 @@ sgl-eval run mmmu_pro \\
{ when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"],
strategy: ["high-throughput"] },
command:
`python3 -m sglang_router.launch_router \\
`env SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests python3 -m sglang_router.launch_router \\
--pd-disaggregation \\
--prefill http://<prefill-host>:{{PREFILL_PORT}} \\
--decode http://<decode-host>:{{DECODE_PORT}} \\
--host 0.0.0.0 --port {{ROUTER_PORT}} \\
--policy consistent_hashing --dp-aware \\
--decode-policy round_robin \\
--decode-policy power_of_two \\
--worker-startup-check-interval 1 \\
--cache-threshold 0.3 \\
--balance-abs-threshold 2 --balance-rel-threshold 1.1 \\
--disable-circuit-breaker --health-failure-threshold 100 \\