diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index 5b91e8242a..428c1d0b2d 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -948,13 +948,13 @@ docker run \ python3 -m sglang.launch_server ``` -The Deployment and Playground Docker outputs use this `v0.5.21` build for the MI355X Pro and Pro Official FP4 multi-node recipes. It is the build the PD roles ran on, and the first one carrying the Aiter MegaMoEv2 kernels the high-throughput prefill role needs. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts. +The Deployment and Playground Docker outputs retain this `v0.5.21` base image for the MI355X Pro and Pro Official FP4 multi-node recipes. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts. For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds. #### Role-based configuration -Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. +Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. The Pro Official commands additionally apply the kernel and routing updates described below. | Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size | |---|---|---|---|---| @@ -966,7 +966,7 @@ These are two separate workers: neither receives `--nnodes 2` or a cross-node TP `--cuda-graph-bs-decode` covers 1 to `--max-running-requests / attn_dp_size`: the full value on the TP-only rows and an eighth on the DP8 rows. Each strategy sets both workers' admission limit and decode's graph sizes together. The TP-only points use `--mem-fraction-static 0.86` and SWA ratio `0.1`; the DP8 points use `0.92` and `0.15`. -All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. Both high-throughput workers enable the FP8 unified KV layout. The commands below show concurrency 256. +All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. All Pro Official PD workers enable `SGLANG_DSV4_UNIFIED_KV_FP8=1`. Low-Latency and Balanced also set `SGLANG_DSV4_DENSE_MXSCALE=1` on both roles; High-Throughput sets mxscale on decode only. The decode dispatch budget is `64 × (DSpark block size + 1)`: `MORI_MAX_DISPATCH_TOKENS_DECODE=448` at concurrency 16 and `256` at concurrency 48 or 256. The commands below show concurrency 256. Select **Pro** to use the original checkpoint's **EAGLE/MTP 3-1-4** head with the same topology and admission limits. Its commands contain no DSpark parameters. In the Playground, **Off (greedy)** removes speculative decoding. Enabling **HiCache** automatically turns off **Unified Cache External Linker** and removes UMBP's socket configuration and optimistic prefill. Enabling the linker turns HiCache off. @@ -1047,6 +1047,7 @@ python3 -m sglang.launch_server \ ```bash Command GPU_MAX_HW_QUEUES=5 \ SGLANG_DSV4_UNIFIED_KV_FP8=1 \ +SGLANG_DSV4_DENSE_MXSCALE=1 \ SGLANG_SHARED_EXPERT_TP1=1 \ SGLANG_DP_SHARED_EXPERT_LOCAL=1 \ SGLANG_DP_USE_GATHERV=1 \ @@ -1102,11 +1103,29 @@ On the prefill env: These recipes leave `MORI_SHMEM_MODE` unset and use MORI's default `STATIC_HEAP`. The pinned Aiter MegaMoEV2 implementation allocates its buffers through `mori_shmem_create_tensor()` / `shmem_malloc()` after SGLang initializes MORI shmem. This requires the symmetric heap; the separate MORI CCO allocator is not used by this implementation. If your launch script exports `MORI_SHMEM_MODE=ISOLATION`, unset it before launching MegaMoE: its peer-pointer lookup requires the heap mapping. Keep `MORI_SHMEM_HEAP_SIZE=8G` for the prefill recipe above. -**Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1`**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way. +**Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1` at every Pro Official PD concurrency**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way. #### Router configuration -Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. Decode holds no reusable prefix, so `--decode-policy round_robin` spreads it evenly. `--dp-aware` takes effect only when the workers run DP attention. +Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. The TP-only presets keep `--decode-policy round_robin`. Pro Official's DP-attention presets (concurrency 192–512, including the High-Throughput preset at 256) instead use: + +```bash Router settings +SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests \ +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill http://:30000 \ + --decode http://:30100 \ + --host 0.0.0.0 --port 8000 \ + --policy consistent_hashing --dp-aware \ + --decode-policy power_of_two \ + --worker-startup-check-interval 1 \ + --cache-threshold 0.3 \ + --balance-abs-threshold 2 --balance-rel-threshold 1.1 \ + --disable-circuit-breaker --health-failure-threshold 100 \ + --health-check-timeout-secs 600 --health-check-interval-secs 30 +``` + +Power-of-two compares in-flight request counts across decode DP ranks. The one-second load polling interval replaces the 30-second default. Check for `Power-of-two load metric: LeastRequests` in the router startup log. The multi-node router presets retain the default request timeout. The standalone high-throughput Playground preset uses `--request-timeout-secs 14400`. At high concurrency the prefill role queues many requests. When the default 1800 s passes the router aborts the request, and an abort that lands while MORI is writing KV cache over RDMA leaves the transfer neither finished nor failed: that prefill worker can no longer send KV cache to decode, and you have to restart it. @@ -1114,6 +1133,34 @@ The multi-node router presets retain the default request timeout. The standalone `--request-timeout-secs` is the timeout of the HTTP request from the router to the worker, not a limit on queue time. At 14400 the router also waits four hours before giving up on a worker that has stopped responding. That is acceptable where prefill queues are long, which is why the lower-concurrency recipes keep the default. Lower it again if you lower the concurrency. +#### Additional Pro Official concurrency points + +The panel keeps its c16 / c48 / c256 strategy presets. Apply the following overrides to the **Multi-Nodes** commands to reproduce the optimized recipe's endpoints. Here `cN` means a target concurrency of N. + +**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 2` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Benchmark with concurrency 1. + +**c32:** start from Low-Latency, keep TP4 → TP8 and block size 6, set both admission limits to 64, and capture decode graph sizes 1–64. Both roles use FP8 KV and mxscale, as at c16 and c48. + +**c192 / c384 / c512:** start from High-Throughput. Keep TP8+DP8+EP8 prefill → TP8+DP8 decode, DSpark block size 3 (4 draft tokens), and decode dispatch budget 256. Set both admission limits to 384 / 768 / 1024 and capture decode graph sizes 1–48 / 1–96 / 1–128, respectively. Keep FP8 KV on both roles, mxscale on decode only, and the power-of-two router settings above. At c384 and c512, add `--request-timeout-secs 14400` to the router. + +**c512 prefill EPLB:** additionally set these environment variables on the prefill worker: + +```bash Prefill environment +export SGLANG_AITER_MEGA_RANK_SYNC=1 +export SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 +export SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 +``` + +Append these flags to its launch command: + +```text Prefill flags +--enable-eplb +--eplb-rebalance-num-iterations 200 +--expert-distribution-recorder-mode stat +``` + +Keep shared-expert fusion enabled. Rank synchronization lets idle DP ranks participate in MegaMoE dispatch; the prefill-only mode records load and advances rebalancing on synchronized extend passes. Do not set `--expert-balancedness-report-mode` with this mode: it can deadlock prefill on the first request. Decode uses TP MoE and receives none of these EPLB settings. EPLB is specific to c512; the c256 panel preset leaves it disabled. + ### 3.8 Unified Cache External Linker (UMBP) The **unified cache external linker** is the second way to move KV cache off the GPU. The linker is the SGLang feature, and you enable it with `--enable-unified-cache-external-linker`. UMBP is one store the linker can use; you select it with `--unified-cache-external-linker-backend mori`. The Playground card uses the feature name, not the store name. diff --git a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index 304a320070..2707045c6b 100644 --- a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -52,6 +52,8 @@ export const config = { // AgentX 1P1D recipe: InferenceX PR #3664, commit df5e433239d7. // The strategy presets use c16 / c48 / c256. Pro uses its MTP head; // Pro Official is the InferenceX checkpoint. Acceptance simulation is omitted. + // Pro Official kernel/routing updates follow the optimized AgentX recipe; + // see section 3.7 for the c1 / c512 overrides. overlayDims: [ { id: "pdMode", title: "PD role", default: "prefill", showWhen: (s) => s.hw === "mi355x" && s.quant === "fp4" @@ -73,13 +75,15 @@ export const config = { const tp = dp || !prefill ? 8 : 4; const image = "lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001"; const router = [ + ...(official && dp ? ["env SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests"] : []), "python3 -m sglang_router.launch_router", " --pd-disaggregation", " --prefill http://:30000", " --decode http://:30100", " --host 0.0.0.0 --port 8000", " --policy consistent_hashing --dp-aware", - " --decode-policy round_robin", + ` --decode-policy ${official && dp ? "power_of_two" : "round_robin"}`, + ...(official && dp ? [" --worker-startup-check-interval 1"] : []), " --cache-threshold 0.3", " --balance-abs-threshold 2 --balance-rel-threshold 1.1", " --disable-circuit-breaker --health-failure-threshold 100", @@ -111,8 +115,12 @@ export const config = { // SGE follows NIC capabilities; MORI already selects 2 on AMD AINIC. `GPU_MAX_HW_QUEUES=${dp ? 5 : 2}`, ...(prefill ? ["UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock"] : []), + // Both PD peers must use the same KV layout. TP prefill and all + // decode points use mxscale; MegaMoE prefill keeps its existing GEMMs. + ...(official || dp ? ["SGLANG_DSV4_UNIFIED_KV_FP8=1"] : []), + ...(official && (!dp || !prefill) ? ["SGLANG_DSV4_DENSE_MXSCALE=1"] : []), + ...(official && !prefill ? [`MORI_MAX_DISPATCH_TOKENS_DECODE=${64 * (gamma + 1)}`] : []), ...(dp ? [ - "SGLANG_DSV4_UNIFIED_KV_FP8=1", // Shared-expert/gatherv default off; reduce-scatter defaults on in ROCm. ...(prefill ? ["SGLANG_DP_USE_REDUCE_SCATTER=0"] : [ "SGLANG_SHARED_EXPERT_TP1=1", @@ -679,6 +687,7 @@ sgl-eval run mmmu_pro \\ { mode: "prefill", when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["low-latency"] }, + env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1"], stripFlags: ["--prefill-decode-interval"], flags: [ "--load-balance-method round_robin", @@ -697,6 +706,8 @@ sgl-eval run mmmu_pro \\ { mode: "decode", when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["low-latency"] }, + env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1", + "MORI_MAX_DISPATCH_TOKENS_DECODE=448"], stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"], flags: [ "--load-balance-method round_robin", @@ -716,6 +727,7 @@ sgl-eval run mmmu_pro \\ { mode: "prefill", when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["balanced"] }, + env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1"], stripFlags: ["--prefill-decode-interval"], flags: [ "--tp 4", @@ -734,6 +746,8 @@ sgl-eval run mmmu_pro \\ { mode: "decode", when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["balanced"] }, + env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1", + "MORI_MAX_DISPATCH_TOKENS_DECODE=256"], stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"], flags: [ "--load-balance-method round_robin", @@ -835,7 +849,8 @@ sgl-eval run mmmu_pro \\ when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["high-throughput"] }, // Same value as prefill — the handshake compares KV layouts. - env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1"], + env: ["SGLANG_DSV4_UNIFIED_KV_FP8=1", "SGLANG_DSV4_DENSE_MXSCALE=1", + "MORI_MAX_DISPATCH_TOKENS_DECODE=256"], stripFlags: ["--prefill-decode-interval", "--chunked-prefill-size"], flags: [ "--load-balance-method round_robin", @@ -876,9 +891,9 @@ sgl-eval run mmmu_pro \\ // MI355X fronts the 1P x 1D agentic pair with a cache-aware router rather // than the default round-robin: consistent_hashing keeps a conversation on // the prefill worker that already holds its prefix (decode holds no - // reusable prefix, so it stays round-robin), and the tight balance - // thresholds stop that affinity from starving the peer at the small - // running-request ceiling the roles above use. Health checking stays + // reusable prefix, so Pro Official DP uses power-of-two; other presets + // keep round-robin). The tight balance thresholds stop that affinity + // from starving the peer at the small running-request ceiling above. Health checking stays // enabled here (unlike the default's 999999s interval) because a long // agentic run should notice a wedged worker, but it is slack enough that // a multi-minute prefill is not mistaken for a failure. @@ -896,13 +911,14 @@ sgl-eval run mmmu_pro \\ { when: { hw: ["mi355x"], variant: ["pro-official"], quant: ["fp4"], strategy: ["high-throughput"] }, command: -`python3 -m sglang_router.launch_router \\ +`env SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests python3 -m sglang_router.launch_router \\ --pd-disaggregation \\ --prefill http://:{{PREFILL_PORT}} \\ --decode http://:{{DECODE_PORT}} \\ --host 0.0.0.0 --port {{ROUTER_PORT}} \\ --policy consistent_hashing --dp-aware \\ - --decode-policy round_robin \\ + --decode-policy power_of_two \\ + --worker-startup-check-interval 1 \\ --cache-threshold 0.3 \\ --balance-abs-threshold 2 --balance-rel-threshold 1.1 \\ --disable-circuit-breaker --health-failure-threshold 100 \\