minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[AMD] Update v4 amd cookbook 0828 (#36828)

Browse Source
This commit is contained in:
Thomas Wang
2026-08-28 17:20:22 -07:00
committed by GitHub
parent 36afd70f9c
commit 89816a21a1
2 changed files with 26 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+24 -4
View File
@@ -52,11 +52,11 @@ docker run --gpus all \
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
For example: For example:
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260822` - **MI355X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260828`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi30x-20260822` - **MI300X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260828`
```bash Command ```bash Command
docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 docker pull lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828
docker run \ docker run \
--device=/dev/kfd --device=/dev/dri \ --device=/dev/kfd --device=/dev/dri \
@@ -66,7 +66,7 @@ docker run \
-p 30000:30000 \ -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \ -v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \ --env "HF_TOKEN=<your-hf-token>" \
lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 \ lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828 \
sglang serve <use args below> sglang serve <use args below>
``` ```
@@ -339,6 +339,26 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. - **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
- **Kernels** — uses the Unified KV attention and the flydsl MoE. - **Kernels** — uses the Unified KV attention and the flydsl MoE.
**MoRI EP (AMD expert parallelism)**
On AMD, expert parallelism uses the **MoRI** all-to-all backend (`--moe-a2a-backend mori`), not DeepEP. Add the flags below on top of the verified recipe when sharding experts across GPUs; set `--ep-size` to the EP degree (typically the GPU count on one node).
Two optional env vars improve MoRI throughput (both off by default):
```bash Command
export SGLANG_MORI_DISPATCH_DTYPE=mxfp8
export SGLANG_MORI_RECV_BOUND=1
sglang serve \
--model-path {MODEL_PATH} \
--tp 8 --dp 8 --enable-dp-attention \
--ep-size 8 --moe-a2a-backend mori --deepep-mode normal \
<other args from the Deploy panel>
```
- **FP4**: enable both env vars.
- **FP8**: use `SGLANG_MORI_RECV_BOUND=1` only; omit `SGLANG_MORI_DISPATCH_DTYPE=mxfp8`.
**MegaMoE** **MegaMoE**
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
+2 -2
View File
@@ -183,8 +183,8 @@ sgl-eval run aime25 \\
gb300: "lmsysorg/sglang:latest", gb300: "lmsysorg/sglang:latest",
// AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you // AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you
// re-verify on a newer build. // re-verify on a newer build.
mi300x: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi30x-20260822", mi300x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260828",
mi355x: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260822", mi355x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260828",
}, },
// Pre-selects the issue template's `model` dropdown on "Submit verified cell". // Pre-selects the issue template's `model` dropdown on "Submit verified cell".