minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[AMD] GLM-5.2 MI355X MXFP4: bump image to 20260916, use HIP Top-K (#40148)

Browse Source
This commit is contained in:
ChangLiu0709
2026-09-18 23:21:58 +08:00
committed by GitHub
parent 7714b182f2
commit 45938a24ae
2 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/GLM/GLM-5.2.mdx
+1 -1
View File
@@ -124,7 +124,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
</Note> </Note>
- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913`. Choose **Low-Latency** for the GPU-resident TP8/EP1 topology, or **High-Throughput** for the TP4/EP4 topology. Both emit `--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` with the Triton DSA prefill and decode backends. The `20260913` image also enables `SGLANG_OPT_USE_TOPK_V2=true` (v2 fused top-k kernel for GLM-5.x on ROCm, landed in [#36684](https://github.com/sgl-project/sglang/pull/36684) and [#36851](https://github.com/sgl-project/sglang/pull/36851)). The TP4/EP4 InferenceX benchmark adds HiCache DRAM offload, but those host-specific knobs are intentionally omitted from the portable cookbook command. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. - **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916`. Choose **Low-Latency** for the GPU-resident TP8/EP1 topology, or **High-Throughput** for the TP4/EP4 topology. Both emit `--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` with the Triton DSA prefill and decode backends. The TP4/EP4 InferenceX benchmark adds HiCache DRAM offload, but those host-specific knobs are intentionally omitted from the portable cookbook command. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions.
## 3. Advanced Usage ## 3. Advanced Usage
docs/src/snippets/configs/zai-org/glm-5.2.jsx
+5 -5
View File
@@ -97,7 +97,7 @@ sgl-eval run aime25 \\
gb300: "lmsysorg/sglang:latest", gb300: "lmsysorg/sglang:latest",
b300: "lmsysorg/sglang:latest", b300: "lmsysorg/sglang:latest",
mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618", mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618",
"mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913", "mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916",
mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
}, },
@@ -1019,7 +1019,7 @@ sgl-eval run aime25 \\
{ {
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "low-latency", nodes: "single" }, match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "low-latency", nodes: "single" },
verified: false, verified: false,
env: ["SGLANG_OPT_USE_TOPK_V2=true"], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
"--model-path {{MODEL_NAME}}", "--model-path {{MODEL_NAME}}",
@@ -1042,7 +1042,7 @@ sgl-eval run aime25 \\
{ {
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "balanced", nodes: "single" }, match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "balanced", nodes: "single" },
verified: false, verified: false,
env: ["SGLANG_OPT_USE_TOPK_V2=true"], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
"--model-path {{MODEL_NAME}}", "--model-path {{MODEL_NAME}}",
@@ -1062,7 +1062,7 @@ sgl-eval run aime25 \\
{ {
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "high-throughput", nodes: "single" }, match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "high-throughput", nodes: "single" },
verified: false, verified: false,
env: ["SGLANG_OPT_USE_TOPK_V2=true"], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
"--model-path {{MODEL_NAME}}", "--model-path {{MODEL_NAME}}",
@@ -1094,7 +1094,7 @@ sgl-eval run aime25 \\
{ {
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "mtp-314", nodes: "single" }, match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "mtp-314", nodes: "single" },
verified: false, verified: false,
env: ["SGLANG_OPT_USE_TOPK_V2=true"], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
"--model-path {{MODEL_NAME}}", "--model-path {{MODEL_NAME}}",