minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[AMD] Update v4 amd cookbook 0903 (#37829)

Browse Source
This commit is contained in:
Thomas Wang
2026-09-03 22:11:13 -07:00
committed by GitHub
parent ca7b8efd87
commit 225129fe44
2 changed files with 65 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+17 -2
View File
@@ -307,7 +307,9 @@ fused shared expert slot.
**FP4 Indexer (Experimental)**
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
On **NVIDIA (SM100 / SM120)**, pair the flag with DeepGEMM FP4 indexer support and the FlashInfer MXFP4 MoE runner:
```bash Command
# Please use the latest main branch for this feature.
@@ -318,6 +320,18 @@ sglang serve \
--enable-deepseek-v4-fp4-indexer
```
On **AMD MI355X (gfx95)**, the AITER FP4 indexer kernels are available on ROCm. Keep the standard MI355 recipe and add only the indexer flag. Do not need to pass `--moe-runner-backend flashinfer_mxfp4`:
```bash Command
# Please use the latest ROCm image for this feature.
sglang serve \
--model-path deepseek-ai/DeepSeek-V4-Pro \
--tp 8 \
--attention-backend dsv4 \
--enable-deepseek-v4-fp4-indexer \
<other args from the Deploy panel>
```
**NVFP4 Hybrid Checkpoints**
The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and
@@ -377,9 +391,10 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`.
- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --enable-two-batch-overlap --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
- **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe.
**MoRI EP (AMD expert parallelism)**
docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+48
View File
@@ -1828,6 +1828,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -1851,6 +1855,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2607,6 +2615,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2639,6 +2651,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2675,6 +2691,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2707,6 +2727,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2774,6 +2798,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2810,6 +2838,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2877,6 +2909,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2913,6 +2949,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -2980,6 +3020,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",
@@ -3016,6 +3060,10 @@ sgl-eval run mmmu_pro \\
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
"--prefill-decode-interval 10",
"--enable-two-batch-overlap",
"--attention-backend dsv4",
"--page-size 256",