[AMD] GLM 5.2 MXFP4 SGLANG COOKBOOK (#34379)
Signed-off-by: Sirra <asirra@amd.com> Co-authored-by: giovanniguastiamd <giovanni.guasti@amd.com>
This commit is contained in:
ajith-sirra-amd
co-authored by
giovanniguastiamd
giovanniguastiamd
parent
00bdafe944
commit
b5d1453ed2
@@ -70,7 +70,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench.
|
||||
**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`).
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
@@ -96,12 +96,17 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/amd/GLM-5.2-MXFP4">GLM-5.2-MXFP4</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · MXFP4</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).
|
||||
|
||||
**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4).
|
||||
**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
@@ -114,6 +119,7 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.
|
||||
|
||||
- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).
|
||||
- **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the same DSA tilelang backends, `--chunked-prefill-size`, and `--watchdog-timeout` as the FP8/BF16 recipes above. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified.
|
||||
|
||||
<Note>
|
||||
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
|
||||
|
||||
@@ -17,6 +17,7 @@ export const config = {
|
||||
{ id: "fp8", label: "FP8" },
|
||||
{ id: "bf16", label: "BF16" },
|
||||
{ id: "nvfp4", label: "NVFP4" },
|
||||
{ id: "mxfp4", label: "MXFP4" },
|
||||
],
|
||||
strategies: [
|
||||
{ id: "low-latency", label: "Low-Latency" },
|
||||
@@ -32,6 +33,7 @@ export const config = {
|
||||
"default|fp8": "zai-org/GLM-5.2-FP8",
|
||||
"default|bf16": "zai-org/GLM-5.2",
|
||||
"default|nvfp4": "nvidia/GLM-5.2-NVFP4",
|
||||
"default|mxfp4": "amd/GLM-5.2-MXFP4",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
@@ -95,6 +97,7 @@ sgl-eval run aime25 \\
|
||||
gb300: "lmsysorg/sglang:latest",
|
||||
b300: "lmsysorg/sglang:latest",
|
||||
mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618",
|
||||
"mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728",
|
||||
mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
|
||||
mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
|
||||
},
|
||||
@@ -182,6 +185,11 @@ sgl-eval run aime25 \\
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"],
|
||||
disable: { hw: ["mi355x", "mi325x", "mi300x"] },
|
||||
disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." },
|
||||
{ id: "mtp-314", label: "EAGLE / MTP 3-1-4 (agentic · MI355X MXFP4)",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"],
|
||||
enable: { hw: ["mi355x"], quant: ["mxfp4"] },
|
||||
enableReason: "Validated on MI355X gfx950 with amd/GLM-5.2-MXFP4 (InferenceX AgentX sweep, GSM8K em_strict 0.971). num-steps=3 stays within the validated gfx950 spec-decode build envelope (≤3). Pair with SGLANG_SIMULATE_ACC_LEN=2.99 for benchmarking (golden AL from golden_al_distribution/glm5.2_mtp.yaml, thinking_on, num_speculative_tokens=3)." },
|
||||
],
|
||||
},
|
||||
|
||||
@@ -994,6 +1002,105 @@ sgl-eval run aime25 \\
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ====================================================================
|
||||
// AMD MI355X + MXFP4 — amd/GLM-5.2-MXFP4 (Quark). TP4: the 4-bit MoE
|
||||
// weights fit a 4-GPU slice, mirroring the amd/GLM-5.1-MXFP4 MI355X recipe (same DSA
|
||||
// architecture family) — --trust-remote-code (Quark custom quant config)
|
||||
// and --kv-cache-dtype fp8_e4m3 both come from that precedent. Pinned to a
|
||||
// newer image (v0.5.16, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16
|
||||
// mi355x cells. MTP (mtp-314, steps=3) is validated on MI355X gfx950 with
|
||||
// this precision — see the mtp-314 cell below. Not yet benchmarked for
|
||||
// GLM-5.2 on the base strategies → verified:false.
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--chunked-prefill-size 131072",
|
||||
"--mem-fraction-static 0.80",
|
||||
"--watchdog-timeout 1200",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--chunked-prefill-size 32768",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--cuda-graph-max-bs 128",
|
||||
"--max-running-requests 80",
|
||||
"--watchdog-timeout 1200",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--cuda-graph-max-bs 256",
|
||||
"--max-running-requests 256",
|
||||
"--watchdog-timeout 1200",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// MI355X + MXFP4 + MTP (mtp-314): validated AgentX recipe.
|
||||
// steps=3 stays within the gfx950 spec-decode build envelope (≤3).
|
||||
// mem-fraction-static 0.80: headroom for MTP draft buffer on top of
|
||||
// 4-bit MoE weights + KV cache (matches InferenceX AgentX harness conc≤16).
|
||||
// For benchmarking: set SGLANG_SIMULATE_ACC_LEN=2.99,
|
||||
// SGLANG_SIMULATE_ACC_METHOD=match-expected,
|
||||
// SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token
|
||||
// (golden AL: golden_al_distribution/glm5.2_mtp.yaml, thinking_on, num_speculative_tokens=3).
|
||||
{
|
||||
match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "mtp-314", nodes: "single" },
|
||||
verified: false,
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--speculative-algorithm EAGLE",
|
||||
"--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1",
|
||||
"--speculative-num-draft-tokens 4",
|
||||
"--chunked-prefill-size 131072",
|
||||
"--mem-fraction-static 0.80",
|
||||
"--cuda-graph-max-bs 160",
|
||||
"--max-running-requests 160",
|
||||
"--watchdog-timeout 1800",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
|
||||
Reference in New Issue
Block a user