[AMD] Add dspark config and agentic workload section for deepseek-v4 model (#39252)
This commit is contained in:
Thomas Wang
@@ -51,11 +51,11 @@ docker run --gpus all \
|
||||
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
|
||||
|
||||
For example:
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260829`
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260911`
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260829
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911
|
||||
|
||||
docker run \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
@@ -65,7 +65,7 @@ docker run \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260829 \
|
||||
lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260911 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
@@ -391,10 +391,11 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
|
||||
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
|
||||
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
|
||||
- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
|
||||
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
|
||||
- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
|
||||
- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Do not use EAGLE on 0813. Enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path.
|
||||
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
|
||||
- **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe.
|
||||
- **Agentic long-context (MI355X Pro FP4)** — for multi-turn trace replay, add `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` on the DP path so a single long prefill does not monopolise the engine.
|
||||
- **Agentic long-context (MI355X Pro Official FP4)** — TP-only serving adds `--prefill-decode-interval 10` for scheduler stability. The DP path additionally needs `--enable-dp-lm-head` (required for DSpark under DP attention), `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` so a single long prefill does not monopolise the engine, plus the flags in the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark).
|
||||
|
||||
**MoRI EP (AMD expert parallelism)**
|
||||
|
||||
@@ -692,7 +693,7 @@ Larger blocks can improve decode latency when acceptance stays high, but they al
|
||||
|
||||
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
|
||||
|
||||
DSpark currently requires CUDA and `pp_size == 1`. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The MI355X Flash Official recipes therefore run target-only, and so do the DP-Attention recipes in the Deploy panel — for a DP-Attention configuration that does run DSpark, see the [agentic recipe below](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
DSpark requires `pp_size == 1`. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The MI355X Flash Official recipes therefore run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
|
||||
### 3.5 Vision (Image Inputs)
|
||||
|
||||
@@ -795,3 +796,94 @@ python3 -m sglang.launch_server \
|
||||
```
|
||||
|
||||
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 6144 tokens per rank.
|
||||
|
||||
### 3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)
|
||||
|
||||
DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7).
|
||||
|
||||
**TP8, concurrency 1–16:**
|
||||
```bash Command
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
|
||||
AITER_BF16_FP8_MOE_BOUND=0 \
|
||||
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--prefill-decode-interval 10 \
|
||||
--attention-backend dsv4 \
|
||||
--enable-deepseek-v4-fp4-indexer \
|
||||
--page-size 256 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--enforce-shared-experts-fusion \
|
||||
--mem-fraction-static 0.86 \
|
||||
--chunked-prefill-size 16384 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--speculative-dspark-block-size 6 \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
|
||||
Concurrency 32–48 keeps the same TP-only command and adds HiCache DRAM offload. DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio). Ratio `1.5` is the value that stays under a ~2.7 TB host DRAM budget at TP8:
|
||||
|
||||
```bash Command
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 1.5 \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-mem-layout page_first_direct
|
||||
```
|
||||
|
||||
**DP8 (DP Attention), concurrency 128–256:**
|
||||
```bash Command
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
|
||||
AITER_BF16_FP8_MOE_BOUND=0 \
|
||||
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
|
||||
SGLANG_SHARED_EXPERT_TP1=1 \
|
||||
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
|
||||
SGLANG_DP_USE_GATHERV=1 \
|
||||
SGLANG_DP_USE_REDUCE_SCATTER=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--enable-dp-lm-head \
|
||||
--enable-prefill-delayer \
|
||||
--enable-dp-attention-local-control-broadcast \
|
||||
--tokenizer-worker-num 8 \
|
||||
--stream-interval 20 \
|
||||
--prefill-decode-interval 10 \
|
||||
--prefill-delayer-token-usage-low-watermark 0.7 \
|
||||
--attention-backend dsv4 \
|
||||
--enable-deepseek-v4-fp4-indexer \
|
||||
--page-size 256 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--enforce-shared-experts-fusion \
|
||||
--mem-fraction-static 0.92 \
|
||||
--chunked-prefill-size 65536 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--speculative-dspark-block-size 6 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 1.5 \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-mem-layout page_first_direct \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
|
||||
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention.
|
||||
|
||||
@@ -221,8 +221,8 @@ sgl-eval run mmmu_pro \\
|
||||
gb300: "lmsysorg/sglang:latest",
|
||||
// AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you
|
||||
// re-verify on a newer build.
|
||||
mi300x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260829",
|
||||
mi355x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829",
|
||||
mi300x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260911",
|
||||
mi355x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911",
|
||||
},
|
||||
|
||||
// Pre-selects the issue template's `model` dropdown on "Submit verified cell".
|
||||
@@ -337,9 +337,9 @@ sgl-eval run mmmu_pro \\
|
||||
hide: { variant: ["flash", "pro"] },
|
||||
disable: [
|
||||
{ when: { dpAttnOn: [true] },
|
||||
reason: "DSpark is not compatible with DP Attention on the current release." },
|
||||
{ when: { hw: ["mi300x", "mi355x"] },
|
||||
reason: "DSpark currently requires CUDA." },
|
||||
reason: "DSpark is not compatible with DP Attention on the current release. For a DP + DSpark agentic recipe, see the cookbook §3.6 (B200) / §3.7 (MI355X) notes." },
|
||||
{ when: { hw: ["mi300x"] },
|
||||
reason: "DSpark on ROCm is documented for MI355X Pro Official (0813); MI300X still requires CUDA." },
|
||||
] },
|
||||
{ id: "ngram", label: "NGRAM",
|
||||
flags: ["--speculative-algorithm NGRAM",
|
||||
@@ -470,6 +470,7 @@ sgl-eval run mmmu_pro \\
|
||||
{ id: "3", label: "3", flags: ["--speculative-dspark-block-size 3"] },
|
||||
{ id: "4", label: "4", flags: ["--speculative-dspark-block-size 4"] },
|
||||
{ id: "5", label: "5", flags: ["--speculative-dspark-block-size 5"] },
|
||||
{ id: "6", label: "6", flags: ["--speculative-dspark-block-size 6"] },
|
||||
],
|
||||
},
|
||||
],
|
||||
@@ -1794,11 +1795,11 @@ sgl-eval run mmmu_pro \\
|
||||
},
|
||||
// ====================================================================
|
||||
// MI355X + FP4 — Pro Official (0813)
|
||||
// Mirrors the verified Pro cells; speculative decoding re-fitted to the
|
||||
// bundled DSpark head. NOT yet run end-to-end on this hardware.
|
||||
// Bundled DSpark head. Low-latency is TP-only + DSPARK; balanced /
|
||||
// high-throughput stay target-only in the Deploy panel (DP Attention).
|
||||
// The DP + DSpark agentic path is documented in cookbook §3.7.
|
||||
// ====================================================================
|
||||
{
|
||||
// DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only.
|
||||
match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"],
|
||||
@@ -1806,19 +1807,22 @@ sgl-eval run mmmu_pro \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--prefill-decode-interval 10",
|
||||
"--attention-backend dsv4",
|
||||
"--enable-deepseek-v4-fp4-indexer",
|
||||
"--page-size 256",
|
||||
"--mem-fraction-static 0.90",
|
||||
"--swa-full-tokens-ratio 0.15",
|
||||
"--enforce-shared-experts-fusion",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--chunked-prefill-size 16384",
|
||||
"--speculative-algorithm DSPARK",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
// DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only.
|
||||
// DSpark + DP Attention is documented in cookbook §3.7, not this cell.
|
||||
match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_SHARED_EXPERT_TP1=1", "SGLANG_DP_SHARED_EXPERT_LOCAL=1", "SGLANG_DP_USE_GATHERV=1", "SGLANG_DP_USE_REDUCE_SCATTER=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"],
|
||||
@@ -1832,8 +1836,8 @@ sgl-eval run mmmu_pro \\
|
||||
"--tokenizer-worker-num 8",
|
||||
"--stream-interval 20",
|
||||
"--prefill-decode-interval 10",
|
||||
"--enable-two-batch-overlap",
|
||||
"--attention-backend dsv4",
|
||||
"--enable-deepseek-v4-fp4-indexer",
|
||||
"--page-size 256",
|
||||
"--mem-fraction-static 0.90",
|
||||
"--swa-full-tokens-ratio 0.15",
|
||||
@@ -1845,7 +1849,7 @@ sgl-eval run mmmu_pro \\
|
||||
],
|
||||
},
|
||||
{
|
||||
// DSpark requires CUDA; EAGLE binds a head that accepts nothing on 0813 -> target-only.
|
||||
// DSpark + DP Attention is documented in cookbook §3.7, not this cell.
|
||||
match: { hw: "mi355x", variant: "pro-official", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
env: ["SGLANG_USE_ROCM700A=0", "TORCH_BLAS_PREFER_HIPBLASLT=1", "SGLANG_SHARED_EXPERT_TP1=1", "SGLANG_DP_SHARED_EXPERT_LOCAL=1", "SGLANG_DP_USE_GATHERV=1", "SGLANG_DP_USE_REDUCE_SCATTER=1", "SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton", "AITER_BF16_FP8_MOE_BOUND=0", "SGLANG_OPT_USE_AITER_BATCHED_GEMM=true"],
|
||||
@@ -1859,8 +1863,8 @@ sgl-eval run mmmu_pro \\
|
||||
"--tokenizer-worker-num 8",
|
||||
"--stream-interval 20",
|
||||
"--prefill-decode-interval 10",
|
||||
"--enable-two-batch-overlap",
|
||||
"--attention-backend dsv4",
|
||||
"--enable-deepseek-v4-fp4-indexer",
|
||||
"--page-size 256",
|
||||
"--mem-fraction-static 0.90",
|
||||
"--swa-full-tokens-ratio 0.15",
|
||||
|
||||
Reference in New Issue
Block a user