[AMD] Update v4 cookbook for megamoe, fp8 kv attn, BCG (#41458)

This commit is contained in:
Thomas Wang
2026-09-27 10:53:21 -07:00
committed by GitHub
parent e581520c67
commit b252aceffe
2 changed files with 77 additions and 9 deletions
@@ -51,11 +51,11 @@ docker run --gpus all \
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
For example:
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260914`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260914`
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi30x-20260926`
```bash Command
docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914
docker pull lmsysorg/sglang-rocm:v0.5.20-rocm720-{mi35x,mi30x}-20260926
docker run \
--device=/dev/kfd --device=/dev/dri \
@@ -65,7 +65,7 @@ docker run \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \
lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914 \
lmsysorg/sglang-rocm:v0.5.20-rocm720-{mi35x,mi30x}-20260926 \
sglang serve <use args below>
```
@@ -877,11 +877,17 @@ Concurrency 256 uses 2P1D — two prefill workers, one node each, added with a s
### 3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)
DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7).
DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). Keep `SGLANG_DSV4_UNIFIED_KV_FP8=1` on every arm.
- **BCG**: conc 1–4 — `--cuda-graph-backend-prefill breakable`
- **Grouped ASM**: TP-only, conc ≥ 16 — `SGLANG_DSV4_GROUPED_ASM=1`
- **EP**: MegaMoE; at concurrency 384 lower `--prefill-decode-interval` to `15`
**TP8, concurrency 1–16:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
@@ -917,9 +923,11 @@ Concurrency 32–48 keeps the same TP-only command and adds HiCache DRAM offload
--hicache-mem-layout page_first_direct
```
**DP8 (DP Attention), concurrency 128–256:**
**DP8 (DP Attention), concurrency 128:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
@@ -942,6 +950,7 @@ python3 -m sglang.launch_server \
--stream-interval 20 \
--prefill-decode-interval 20 \
--prefill-delayer-token-usage-low-watermark 0.7 \
--load-balance-method total_requests \
--attention-backend dsv4 \
--enable-deepseek-v4-fp4-indexer \
--page-size 256 \
@@ -964,6 +973,65 @@ python3 -m sglang.launch_server \
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware` so multi-turn sessions land on the rank holding the longest radix/HiCache prefix; above concurrency 160 add `--balance-abs-threshold 32`.
**DP8 EP8 MegaMoE, concurrency 128–384:**
```bash Command
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=8192 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=17179869184 \
SGLANG_AITER_MEGA_RANK_SYNC=1 \
SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \
SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
GPU_MAX_HW_QUEUES=5 \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--ep 8 \
--enable-dp-attention \
--enable-dp-lm-head \
--enable-prefill-delayer \
--enable-dp-attention-local-control-broadcast \
--tokenizer-worker-num 8 \
--stream-interval 20 \
--prefill-decode-interval 20 \
--prefill-delayer-token-usage-low-watermark 0.7 \
--load-balance-method total_requests \
--attention-backend dsv4 \
--enable-deepseek-v4-fp4-indexer \
--page-size 256 \
--swa-full-tokens-ratio 0.1 \
--kv-cache-dtype fp8_e4m3 \
--moe-a2a-backend megamoe \
--moe-dense-tp-size 1 \
--disable-shared-experts-fusion \
--enable-eplb \
--eplb-rebalance-num-iterations 200 \
--expert-distribution-recorder-mode stat \
--mem-fraction-static 0.85 \
--chunked-prefill-size 65536 \
--tool-call-parser deepseekv4 \
--reasoning-parser deepseek-v4 \
--speculative-algorithm DSPARK \
--speculative-dspark-block-size 6 \
--enable-hierarchical-cache \
--hicache-ratio 1.5 \
--hicache-write-policy write_through \
--hicache-io-backend direct \
--hicache-mem-layout page_first_direct \
--watchdog-timeout 3600
```
### 3.8 PD Disaggregation on MI355X (MORI-IO)
On ROCm the prefill and decode roles talk over **MORI-IO** rather than Mooncake or NiXL, so the **PD Disagg** card in the [Playground above](#playground) hides the CUDA-only transports once MI300X / MI355X is selected and defaults the IB device list to the node's `rdmaN` NICs instead of ConnectX `mlx5_*` names.
@@ -221,10 +221,10 @@ sgl-eval run mmmu_pro \\
gb300: "lmsysorg/sglang:latest",
// AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you
// re-verify on a newer build.
// Pro Official's DSpark PD + UMBP pairs ran end-to-end on this build.
// Pro Official agentic + DSpark PD + UMBP pairs ran end-to-end on this build.
"mi355x|pro-official|fp4": "lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260923",
mi300x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260914",
mi355x: "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260914",
mi300x: "lmsysorg/sglang-rocm:v0.5.20-rocm720-mi30x-20260926",
mi355x: "lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926",
},
// Pre-selects the issue template's `model` dropdown on "Submit verified cell".