minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[AMD] Update v4 amd cookbook (#28423)

Browse Source
This commit is contained in:
Thomas Wang
2026-06-16 18:15:16 -07:00
committed by GitHub
parent 37ef295c78
commit 0d651e653b
4 changed files with 799 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+34 -4
View File
@@ -28,15 +28,15 @@ Then run the **Python** output of the command panel below in that environment.
<Tab title="Docker">
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
**NVIDIA GPUs**
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200). For **RTX PRO 6000 (SM120)**, use the nightly `lmsysorg/sglang:dev` instead — SM120 support isn't in `:latest` yet (see the RTX PRO 6000 note below).
```bash Command
docker pull lmsysorg/sglang:latest
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
```bash Command
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
@@ -47,6 +47,28 @@ docker run --gpus all \
sglang serve <use args below>
```
**AMD GPUs (ROCm)**
AMD uses the daily-updated `lmsysorg/sglang-rocm` images:
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260615`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi30x-20260615`
```bash Command
docker pull lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260615
docker run \
--device=/dev/kfd --device=/dev/dri \
--group-add video \
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
--shm-size 32g --ipc=host \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \
lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260615 \
sglang serve <use args below>
```
</Tab>
</Tabs>
@@ -233,6 +255,14 @@ RTX PRO 6000 (96 GB) runs **Flash only** — V4-Pro doesn't fit on 8× 96 GB. It
`--mem-fraction-static 0.70`; the Deploy panel greys out the other recipes for this card.
HiCache and MegaMoE are **not** supported on RTX PRO 6000. For Docker, use the nightly `lmsysorg/sglang:dev` image — SM120 support isn't in `lmsysorg/sglang:latest` yet (the Deploy panel's Docker mode already points this card at `:dev`).
**AMD (MI300X / MI355X) note**
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`.
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
**MegaMoE**
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput