--- title: GLM-5.2 description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X." --- ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: - **Low-Latency** — fastest reply for a single user. Pick for chat. - **Balanced** — good speed with several users at once. Use for typical multi-user serving. - **High-Throughput** — most tokens per second across many users. Best for batch jobs. import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/zai-org/glm-5.2.jsx"; import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.2-benchmarks.jsx"; All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2. Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc` (H200 FP8 cells: `v0.5.14 @ 49e384ce`). Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5; FP8 balanced 1-1-2 = 2; NVFP4 balanced 2-1-3 = 2); high-throughput has no spec. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. import { Playground } from "/src/snippets/_playground.jsx"; ## 1. Model Introduction **GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`).
Model Architecture Context
GLM-5.2-FP8 MoE · DSA · 256 experts (top-8) · MTP · FP8 1,048,576
GLM-5.2 MoE · DSA · 256 experts (top-8) · MTP · BF16 1,048,576
GLM-5.2-NVFP4 MoE · DSA · 256 experts (top-8) · MTP · NVFP4 1,048,576
GLM-5.2-MXFP4 MoE · DSA · 256 experts (top-8) · MTP · MXFP4 1,048,576
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). **Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4). ## 2. Configuration Tips - **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. - **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute. - **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). - **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. - **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points. - **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. - **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. - **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). - **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the DSA triton backends (`--dsa-prefill-backend triton --dsa-decode-backend triton`, also SGLang's ROCm default), and the same `--chunked-prefill-size` and `--watchdog-timeout` as the FP8/BF16 recipes above. At `tp=4` this shape — 16 query heads per rank on an FP8 KV cache — is the one the gfx950 sparse-MLA kernels are tuned for; the `tp=8` recipes above stay on tilelang, which is what their published numbers were measured with. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). - **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260920`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. ## 3. Advanced Usage ### 3.1 Reasoning GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`). **Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: ` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`: | `reasoning_effort` | Injected system line | Effect | |---|---|---| | *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** | | `"high"` | `Reasoning Effort: High` | dials reasoning **down** | | `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) | ```python Example from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240. 2. **Convert the percentage to a decimal:** 15% = 0.15 3. **Multiply by the total:** 0.15 * 240 = 36 (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.) Answer: 15% of 240 is **36**. Here is how you can calculate it: 0.15 × 240 = 36 ``` ### 3.2 Tool Calling Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `………` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields. ```python Example from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city. Tool calls: [ { "id": "call_13fcd52146934b7781d06d4a", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"} } ] ``` ### 3.3 HiCache (Hierarchical KV Caching) For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. ### 3.4 Claude Code Integration GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: ```bash Command export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" export API_TIMEOUT_MS="3000000" export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" claude ``` Two of these matter specifically for GLM-5.2: - **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. - **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). ### 3.5 Context Parallelism Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below. Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments: ```bash --attn-cp-size 8 \ --enable-prefill-cp \ --cp-strategy interleave \ ``` which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill. When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with ```bash --enable-dsa-cache-layer-split \ --attn-cp-size 8 \ --cp-strategy interleave \ ``` With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. ### 3.6 Agentic Long-Context with HiCache DRAM Offload (NVFP4, MTP) **B300 (TP8):** ```bash Command python3 -m sglang.launch_server \ --model-path nvidia/GLM-5.2-NVFP4 \ --trust-remote-code \ --tp 8 \ --ep-size 1 \ --quantization modelopt_fp4 \ --kv-cache-dtype fp8_e4m3 \ --bf16-gemm-backend cutedsl \ --max-prefill-tokens 8192 \ --chunked-prefill-size 8192 \ --mem-fraction-static 0.85 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-hierarchical-cache \ --hicache-size 270 \ --hicache-write-policy write_back \ --hicache-io-backend direct \ --hicache-mem-layout page_first_direct ``` **B200 (TP8):** same command, with `--mem-fraction-static 0.83` and `--hicache-size 169`. Lower concurrency (c1/c4/c8) uses `--hicache-ratio 0.75` instead.