diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx index 640414a2f..778dbcd6f 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx @@ -103,6 +103,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; - **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. - **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work. +- **DFlash2 (block-diffusion draft).** The **Speculative** card in the [Playground above](#playground) also offers **DFlash2**, which replaces the in-checkpoint MTP layer with the separately trained block-diffusion drafter [`incoai/GLM-5.3-DFlash2`](https://huggingface.co/incoai/GLM-5.3-DFlash2). It proposes a whole block per step and the target verifies the block in one forward pass, so output quality stays the target's. The block size — 8, i.e. 7 draft tokens per verification step — comes from the draft checkpoint's own `dflash_config`, so no `--speculative-num-draft-tokens` is passed; the draft is a small dense model and runs on `fa4` instead of the target's DSA backends. Two prerequisites: the DFlash2 drafter ([PR #35371](https://github.com/sgl-project/sglang/pull/35371)) merged **after v0.5.18**, so install SGLang from `main` (or use a nightly image) rather than the release this page pins; and DFLASH runs on **CUDA/NPU only** and rejects **DP-Attention**, so turn DP-Attention off in the **Attention** card before selecting it on a high-throughput base. The draft repository is public but licensed CC BY-NC-ND 4.0 for research and evaluation. - **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). - **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. - **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. diff --git a/docs/src/snippets/configs/zai-org/glm-5.3.jsx b/docs/src/snippets/configs/zai-org/glm-5.3.jsx index b8b94165a..16786241d 100644 --- a/docs/src/snippets/configs/zai-org/glm-5.3.jsx +++ b/docs/src/snippets/configs/zai-org/glm-5.3.jsx @@ -169,7 +169,8 @@ sgl-eval run aime25 \\ // ----- Card 4: "Speculative Decoding" ----- // GLM-5.3 ships a single MTP (nextn) layer; index_share_for_mtp_iteration reuses the - // DSA indexer topk across draft steps (topk==1 only). + // DSA indexer topk across draft steps (topk==1 only). DFlash2 is the one + // algorithm no Deploy cell ships: its draft is a separate checkpoint. speculative: { options: [ { id: "current", label: "Inherited from base" }, @@ -184,6 +185,25 @@ sgl-eval run aime25 \\ "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"], disable: { hw: ["mi355x", "mi325x", "mi300x"] }, disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, + { id: "dflash", label: "DFlash2 (block diffusion)", + // Block-wise draft from a separate checkpoint, not the in-checkpoint + // MTP layer. The block size (8) comes from the draft's own + // dflash_config, so no --speculative-num-draft-tokens here. The draft + // is a small dense model and does not run on the target's DSA + // backends, hence the explicit draft attention backend. + flags: ["--speculative-algorithm DFLASH", + "--speculative-draft-model-path incoai/GLM-5.3-DFlash2", + "--speculative-draft-attention-backend fa4"], + // The DFlash2 drafter (PR #35371) merged after v0.5.18, so neither the + // release wheel nor the lmsysorg/sglang:latest image this page pins + // carries it. Drop this note once a release ships it. + note: "⚠️ Needs a nightly image: the DFlash2 drafter (PR #35371) is not in the release wheel nor the lmsysorg/sglang:latest image this page pins — install SGLang from main or use a lmsysorg/sglang:dev image. The draft is a separate checkpoint, so fetch incoai/GLM-5.3-DFlash2 alongside the target; it is public but licensed CC BY-NC-ND 4.0 for research and evaluation.", + disable: [ + { when: { dpAttnOn: [true] }, + reason: "DFLASH speculative decoding does not support DP-Attention — the server rejects the combination at startup. Turn DP-Attention off in the Attention card above (the high-throughput recipes enable it)." }, + { when: { hw: ["mi355x", "mi325x", "mi300x"] }, + reason: "DFLASH speculative decoding only supports CUDA and NPU devices; the server rejects it on ROCm at startup." }, + ] }, ], },