diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index f19b39be2..7e507211b 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -52,11 +52,11 @@ docker run --gpus all \ AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. For example: -- **MI355X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260822` -- **MI300X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi30x-20260822` +- **MI355X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260828` +- **MI300X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260828` ```bash Command -docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 +docker pull lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828 docker run \ --device=/dev/kfd --device=/dev/dri \ @@ -66,7 +66,7 @@ docker run \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ - lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 \ + lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828 \ sglang serve ``` @@ -339,6 +339,26 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000. - **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. - **Kernels** — uses the Unified KV attention and the flydsl MoE. +**MoRI EP (AMD expert parallelism)** + +On AMD, expert parallelism uses the **MoRI** all-to-all backend (`--moe-a2a-backend mori`), not DeepEP. Add the flags below on top of the verified recipe when sharding experts across GPUs; set `--ep-size` to the EP degree (typically the GPU count on one node). + +Two optional env vars improve MoRI throughput (both off by default): + +```bash Command +export SGLANG_MORI_DISPATCH_DTYPE=mxfp8 +export SGLANG_MORI_RECV_BOUND=1 + +sglang serve \ + --model-path {MODEL_PATH} \ + --tp 8 --dp 8 --enable-dp-attention \ + --ep-size 8 --moe-a2a-backend mori --deepep-mode normal \ + +``` + +- **FP4**: enable both env vars. +- **FP8**: use `SGLANG_MORI_RECV_BOUND=1` only; omit `SGLANG_MORI_DISPATCH_DTYPE=mxfp8`. + **MegaMoE** MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput diff --git a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index f12d86575..9792c2621 100644 --- a/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -183,8 +183,8 @@ sgl-eval run aime25 \\ gb300: "lmsysorg/sglang:latest", // AMD daily-updated lmsysorg/sglang-rocm images. Bump the dated tag when you // re-verify on a newer build. - mi300x: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi30x-20260822", - mi355x: "lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260822", + mi300x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260828", + mi355x: "lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260828", }, // Pre-selects the issue template's `model` dropdown on "Submit verified cell".