From c415f977b848aef32c2902cfe62bfd99b69c1e30 Mon Sep 17 00:00:00 2001 From: Thomas Wang Date: Thu, 10 Sep 2026 12:41:05 +0800 Subject: [PATCH] [AMD] Update v4 args for agentic workload (#38677) --- docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx | 3 ++- docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index 06420713a..71fb0a706 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -391,10 +391,11 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000. - **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`. - **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node. -- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --enable-two-batch-overlap --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`. +- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`. - **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. - **Kernels** — uses the Unified KV attention and the flydsl MoE. - **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe. +- **Agentic long-context (MI355X Pro FP4)** — for multi-turn trace replay, add `--enable-prefill-delayer --prefill-delayer-token-usage-low-watermark 0.7` on the DP path so a single long prefill does not monopolise the engine. **MoRI EP (AMD expert parallelism)** diff --git a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index 2c2230755..d52040e98 100644 --- a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -2875,6 +2875,7 @@ sgl-eval run mmmu_pro \\ "--model-path {{MODEL_NAME}}", "--tp 8", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15", @@ -2913,8 +2914,8 @@ sgl-eval run mmmu_pro \\ "--tokenizer-worker-num 8", "--stream-interval 20", "--prefill-decode-interval 10", - "--enable-two-batch-overlap", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15", @@ -2953,8 +2954,8 @@ sgl-eval run mmmu_pro \\ "--tokenizer-worker-num 8", "--stream-interval 20", "--prefill-decode-interval 10", - "--enable-two-batch-overlap", "--attention-backend dsv4", + "--enable-deepseek-v4-fp4-indexer", "--page-size 256", "--mem-fraction-static 0.90", "--swa-full-tokens-ratio 0.15",