[AMD] Document GLM-5.2 MXFP4 recipe update on MI355X (#39230)
This commit is contained in:
Zhang, Jiejing
@@ -124,7 +124,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
|
||||
</Note>
|
||||
|
||||
- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.
|
||||
- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260910`. Choose **Low-Latency** for the GPU-resident TP8/EP1 topology, or **High-Throughput** for the TP4/EP4 topology. Both emit `--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` with the Triton DSA prefill and decode backends. The TP4/EP4 InferenceX benchmark adds HiCache DRAM offload, but those host-specific knobs are intentionally omitted from the portable cookbook command. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
|
||||
@@ -97,7 +97,7 @@ sgl-eval run aime25 \\
|
||||
gb300: "lmsysorg/sglang:latest",
|
||||
b300: "lmsysorg/sglang:latest",
|
||||
mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618",
|
||||
"mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728",
|
||||
"mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260910",
|
||||
mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
|
||||
mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
|
||||
},
|
||||
@@ -178,8 +178,12 @@ sgl-eval run aime25 \\
|
||||
{ id: "mtp-516", label: "EAGLE / MTP 5-1-6 (low-latency)",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 5",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6"],
|
||||
disable: { hw: ["mi355x", "mi325x", "mi300x"] },
|
||||
disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." },
|
||||
disable: [
|
||||
{ when: { hw: ["mi300x", "mi325x"] },
|
||||
reason: "MTP/EAGLE speculative decoding is not yet validated for GLM-5.2 on MI300X or MI325X." },
|
||||
{ when: { hw: ["mi355x"], quant: ["fp8", "bf16", "nvfp4"] },
|
||||
reason: "The five-step MI355X recipe is validated only with amd/GLM-5.2-MXFP4." },
|
||||
] },
|
||||
{ id: "mtp-112", label: "EAGLE / MTP 1-1-2 (balanced)",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"],
|
||||
@@ -1007,13 +1011,10 @@ sgl-eval run aime25 \\
|
||||
// weights fit a 4-GPU slice, mirroring the amd/GLM-5.1-MXFP4 MI355X recipe (same DSA
|
||||
// architecture family) — --trust-remote-code (Quark custom quant config)
|
||||
// and --kv-cache-dtype fp8_e4m3 both come from that precedent. Pinned to a
|
||||
// newer image (v0.5.16, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16
|
||||
// mi355x cells. MTP (mtp-314, steps=3) is validated on MI355X gfx950 with
|
||||
// this precision — see the mtp-314 cell below. Not yet benchmarked for
|
||||
// GLM-5.2 on the base strategies → verified:false.
|
||||
// DSA backend: triton (SGLang's ROCm default). At TP4 this shape hits the
|
||||
// gfx950 FP8 sparse-MLA tuning; the TP8 FP8/BF16 cells above stay on
|
||||
// tilelang, where their published numbers were measured.
|
||||
// newer image (v0.5.19, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16
|
||||
// mi355x cells. Low-Latency uses validated TP8/EP1; High-Throughput uses
|
||||
// validated TP4/EP4. Both use five-step MTP from InferenceX PR #2900.
|
||||
// DSA backend: triton (SGLang's ROCm default).
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "low-latency", nodes: "single" },
|
||||
@@ -1022,10 +1023,15 @@ sgl-eval run aime25 \\
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--tp 8",
|
||||
"--ep-size 1",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend triton",
|
||||
"--dsa-decode-backend triton",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 5",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 6",
|
||||
"--chunked-prefill-size 131072",
|
||||
"--mem-fraction-static 0.80",
|
||||
"--watchdog-timeout 1200",
|
||||
@@ -1061,9 +1067,14 @@ sgl-eval run aime25 \\
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--ep-size 4",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend triton",
|
||||
"--dsa-decode-backend triton",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 5",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 6",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--cuda-graph-max-bs-decode 256",
|
||||
"--max-running-requests 256",
|
||||
|
||||
Reference in New Issue
Block a user