From 66f19f5c468f11a75d09a31ccde7b3ac3fd6814d Mon Sep 17 00:00:00 2001 From: "Zhang, Jiejing" Date: Mon, 21 Sep 2026 15:28:27 -0700 Subject: [PATCH] [AMD] Enable HiCache for GLM-5.2 MI355X throughput recipe (#40570) --- docs/cookbook/autoregressive/GLM/GLM-5.2.mdx | 2 +- docs/src/snippets/configs/zai-org/glm-5.2.jsx | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx index a3d8a19dc..7953cc30f 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -124,7 +124,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). -- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916`. Choose **Low-Latency** for the GPU-resident TP8/EP1 topology, or **High-Throughput** for the TP4/EP4 topology. Both emit `--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` with the Triton DSA prefill and decode backends. The TP4/EP4 InferenceX benchmark adds HiCache DRAM offload, but those host-specific knobs are intentionally omitted from the portable cookbook command. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. +- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260920`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. ## 3. Advanced Usage diff --git a/docs/src/snippets/configs/zai-org/glm-5.2.jsx b/docs/src/snippets/configs/zai-org/glm-5.2.jsx index ddd93c874..58e6353c3 100644 --- a/docs/src/snippets/configs/zai-org/glm-5.2.jsx +++ b/docs/src/snippets/configs/zai-org/glm-5.2.jsx @@ -97,7 +97,7 @@ sgl-eval run aime25 \\ gb300: "lmsysorg/sglang:latest", b300: "lmsysorg/sglang:latest", mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618", - "mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916", + "mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260920", mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", }, @@ -1011,7 +1011,7 @@ sgl-eval run aime25 \\ // weights fit a 4-GPU slice, mirroring the amd/GLM-5.1-MXFP4 MI355X recipe (same DSA // architecture family) — --trust-remote-code (Quark custom quant config) // and --kv-cache-dtype fp8_e4m3 both come from that precedent. Pinned to a - // newer image (v0.5.19, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16 + // newer image (v0.5.20, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16 // mi355x cells. Low-Latency uses validated TP8/EP1; High-Throughput uses // validated TP4/EP4. Both use five-step MTP from InferenceX PR #2900. // DSA backend: triton (SGLang's ROCm default). @@ -1075,6 +1075,8 @@ sgl-eval run aime25 \\ "--speculative-num-steps 5", "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6", + "--enable-hierarchical-cache", + "--hicache-ratio 1.0", "--mem-fraction-static 0.85", "--cuda-graph-max-bs-decode 256", "--max-running-requests 256",