From ec5fba5777bf2b7bfaf912d050b21b32df57b8f9 Mon Sep 17 00:00:00 2001 From: Thomas Wang Date: Sun, 13 Sep 2026 10:32:01 +0800 Subject: [PATCH] [AMD] Add dspark config and agentic workload section for deepseek-v4 model (#39252) --- .../autoregressive/DeepSeek/DeepSeek-V4.mdx | 106 ++++++++++++++++-- .../configs/deepseek-ai/deepseek-v4.jsx | 28 +++-- 2 files changed, 115 insertions(+), 19 deletions(-) diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index bdab2b9ad..db749f182 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -51,11 +51,11 @@ docker run --gpus all \ AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. For example: -- **MI355X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829` -- **MI300X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260829` +- **MI355X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911` +- **MI300X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260911` ```bash Command -docker pull lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260829 +docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911 docker run \ --device=/dev/kfd --device=/dev/dri \ @@ -65,7 +65,7 @@ docker run \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ - lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260829 \ + lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911 \ sglang serve ``` @@ -391,10 +391,11 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000. - **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`. - **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node. - **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`. -- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. +- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. +- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Do not use EAGLE on 0813. Enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. - **Kernels** — uses the Unified KV attention and the flydsl MoE. - **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe. -- **Agentic long-context (MI355X Pro FP4)** — for multi-turn trace replay, add `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` on the DP path so a single long prefill does not monopolise the engine. +- **Agentic long-context (MI355X Pro Official FP4)** — TP-only serving adds `--prefill-decode-interval 10` for scheduler stability. The DP path additionally needs `--enable-dp-lm-head` (required for DSpark under DP attention), `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` so a single long prefill does not monopolise the engine, plus the flags in the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). **MoRI EP (AMD expert parallelism)** @@ -692,7 +693,7 @@ Larger blocks can improve decode latency when acceptance stays high, but they al For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override. -DSpark currently requires CUDA and `pp_size == 1`. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The MI355X Flash Official recipes therefore run target-only, and so do the DP-Attention recipes in the Deploy panel — for a DP-Attention configuration that does run DSpark, see the [agentic recipe below](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. +DSpark requires `pp_size == 1`. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The MI355X Flash Official recipes therefore run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. ### 3.5 Vision (Image Inputs) @@ -795,3 +796,94 @@ python3 -m sglang.launch_server \ ``` `--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 6144 tokens per rank. + +### 3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark) + +DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). + +**TP8, concurrency 1–16:** +```bash Command +SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ +SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ +SGLANG_USE_ROCM700A=0 \ +TORCH_BLAS_PREFER_HIPBLASLT=1 \ +SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ +AITER_BF16_FP8_MOE_BOUND=0 \ +SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --prefill-decode-interval 10 \ + --attention-backend dsv4 \ + --enable-deepseek-v4-fp4-indexer \ + --page-size 256 \ + --swa-full-tokens-ratio 0.1 \ + --kv-cache-dtype fp8_e4m3 \ + --enforce-shared-experts-fusion \ + --mem-fraction-static 0.86 \ + --chunked-prefill-size 16384 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --watchdog-timeout 3600 +``` + +Concurrency 32–48 keeps the same TP-only command and adds HiCache DRAM offload. DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio). Ratio `1.5` is the value that stays under a ~2.7 TB host DRAM budget at TP8: + +```bash Command + --enable-hierarchical-cache \ + --hicache-ratio 1.5 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct +``` + +**DP8 (DP Attention), concurrency 128–256:** +```bash Command +SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ +SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ +SGLANG_USE_ROCM700A=0 \ +TORCH_BLAS_PREFER_HIPBLASLT=1 \ +SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ +AITER_BF16_FP8_MOE_BOUND=0 \ +SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ +SGLANG_SHARED_EXPERT_TP1=1 \ +SGLANG_DP_SHARED_EXPERT_LOCAL=1 \ +SGLANG_DP_USE_GATHERV=1 \ +SGLANG_DP_USE_REDUCE_SCATTER=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --dp 8 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --enable-prefill-delayer \ + --enable-dp-attention-local-control-broadcast \ + --tokenizer-worker-num 8 \ + --stream-interval 20 \ + --prefill-decode-interval 10 \ + --prefill-delayer-token-usage-low-watermark 0.7 \ + --attention-backend dsv4 \ + --enable-deepseek-v4-fp4-indexer \ + --page-size 256 \ + --swa-full-tokens-ratio 0.1 \ + --kv-cache-dtype fp8_e4m3 \ + --enforce-shared-experts-fusion \ + --mem-fraction-static 0.92 \ + --chunked-prefill-size 65536 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --enable-hierarchical-cache \ + --hicache-ratio 1.5 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct \ + --watchdog-timeout 3600 +``` + +`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. diff --git a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index d52040e98..7d7904c03 100644 --- a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -221,8 +221,8 @@ sgl-eval run mmmu_pro \\ gb300: "lmsysorg/sglang:latest", // AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you // re-verify on a newer build. - mi300x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260829", - mi355x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829", + mi300x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260911", + mi355x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911", }, // Pre-selects the issue template's `model` dropdown on "Submit verified cell". @@ -337,9 +337,9 @@ sgl-eval run mmmu_pro \\ hide: { variant: ["flash", "pro"] }, disable: [ { when: { dpAttnOn: [true] }, - reason: "DSpark is not compatible with DP Attention on the current release." }, - { when: { hw: ["mi300x", "mi355x"] }, - reason: "DSpark currently requires CUDA." }, + reason: "DSpark is not compatible with DP Attention on the current release. For a DP + DSpark agentic recipe, see the cookbook §3.6 (B200) / §3.7 (MI355X) notes." }, + { when: { hw: ["mi300x"] }, + reason: "DSpark on ROCm is documented for MI355X Pro Official (0813); MI300X still requires CUDA." }, ] }, { id: "ngram", label: "NGRAM", flags: ["--speculative-algorithm NGRAM", @@ -470,6 +470,7 @@ sgl-eval run mmmu_pro \\ { id: "3", label: "3", flags: ["--speculative-dspark-block-size 3"] }, { id: "4", label: "4", flags: ["--speculative-dspark-block-size 4"] }, { id: "5", label: "5", flags: ["--speculative-dspark-block-size 5"] }, + { id: "6", label: "6", flags: ["--speculative-dspark-block-size 6"] }, ], }, ], @@ -1794,11 +1795,11 @@ sgl-eval run mmmu_pro \\ }, // ==================================================================== // MI355X + FP4 — Pro Official (0813) - // Mirrors the verified Pro cells; speculative decoding re-fitted to the - // bundled DSpark head. NOT yet run end-to-end on this hardware. + // Bundled DSpark head. Low-latency is TP-only + DSPARK; balanced / + // high-throughput stay target-only in the Deploy panel (DP Attention). + // The DP + DSpark agentic path is documented in cookbook §3.7. // ==================================================================== { - // DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only. match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "low-latency", nodes: "single" }, verified: false, env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"], @@ -1806,19 +1807,22 @@ sgl-eval run mmmu_pro \\ "--trust-remote-code", "--model-path {{MODEL_NAME}}", "--tp 8", + "--prefill-decode-interval 10", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15", "--enforce-shared-experts-fusion", "--kv-cache-dtype fp8_e4m3", "--chunked-prefill-size 16384", + "--speculative-algorithm DSPARK", "--host {{HOST_IP}}", "--port {{PORT}}", ], }, { - // DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only. + // DSpark + DP Attention is documented in cookbook §3.7, not this cell. match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "balanced", nodes: "single" }, verified: false, env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_SHARED_EXPERT_TP1=1", "SGLANG_DP_SHARED_EXPERT_LOCAL=1", "SGLANG_DP_USE_GATHERV=1", "SGLANG_DP_USE_REDUCE_SCATTER=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"], @@ -1832,8 +1836,8 @@ sgl-eval run mmmu_pro \\ "--tokenizer-worker-num 8", "--stream-interval 20", "--prefill-decode-interval 10", - "--enable-two-batch-overlap", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15", @@ -1845,7 +1849,7 @@ sgl-eval run mmmu_pro \\ ], }, { - // DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only. + // DSpark + DP Attention is documented in cookbook §3.7, not this cell. match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "high-throughput", nodes: "single" }, verified: false, env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_SHARED_EXPERT_TP1=1", "SGLANG_DP_SHARED_EXPERT_LOCAL=1", "SGLANG_DP_USE_GATHERV=1", "SGLANG_DP_USE_REDUCE_SCATTER=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"], @@ -1859,8 +1863,8 @@ sgl-eval run mmmu_pro \\ "--tokenizer-worker-num 8", "--stream-interval 20", "--prefill-decode-interval 10", - "--enable-two-batch-overlap", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15",