diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx index e515d73c6..3b2444cfc 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -70,7 +70,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; ## 1. Model Introduction -**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. +**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`). @@ -96,12 +96,17 @@ import { Playground } from "/src/snippets/_playground.jsx"; + + + + +
MoE · DSA · 256 experts (top-8) · MTP · NVFP4 1,048,576
GLM-5.2-MXFP4MoE · DSA · 256 experts (top-8) · MTP · MXFP41,048,576
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). -**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4). +**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4). ## 2. Configuration Tips @@ -114,6 +119,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; - **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. - **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). +- **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the same DSA tilelang backends, `--chunked-prefill-size`, and `--watchdog-timeout` as the FP8/BF16 recipes above. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). diff --git a/docs/src/snippets/configs/zai-org/glm-5.2.jsx b/docs/src/snippets/configs/zai-org/glm-5.2.jsx index 945fa2a03..d5ee97d6b 100644 --- a/docs/src/snippets/configs/zai-org/glm-5.2.jsx +++ b/docs/src/snippets/configs/zai-org/glm-5.2.jsx @@ -1,1151 +1,1258 @@ -// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration). -// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr`/`--host`/`--port` literals — engine injects them. - -export const config = { - modelName: "GLM-5.2", - - supportedHardware: [ - "h200", "b200", "gb300", "b300", - "mi355x", "mi325x", "mi300x", - ], - - // Single released checkpoint — no size/mode split. - variants: [ - { id: "default", label: "GLM-5.2", subtitle: "MoE · DSA" }, - ], - quantizations: [ - { id: "fp8", label: "FP8" }, - { id: "bf16", label: "BF16" }, - { id: "nvfp4", label: "NVFP4" }, - ], - strategies: [ - { id: "low-latency", label: "Low-Latency" }, - { id: "balanced", label: "Balanced" }, - { id: "high-throughput", label: "High-Throughput" }, - ], - nodesOptions: [ - { id: "single", label: "Single Node" }, - { id: "multi-2", label: "Multi-Nodes" }, - ], - - modelNames: { - "default|fp8": "zai-org/GLM-5.2-FP8", - "default|bf16": "zai-org/GLM-5.2", - "default|nvfp4": "nvidia/GLM-5.2-NVFP4", - }, - - placeholders: { - HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, - PORT: { target: "command", label: "Bind port", default: "30000" }, - NODE0_IP: { target: "command", label: "Head node IP", default: "" }, - NODE_RANK: { target: "command", label: "This node rank", default: "" }, - CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, - CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, - }, - - curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ --H 'Content-Type: application/json' \\ --d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, - - // Reproduce commands for the Benchmark card's "⚡ Reproduce" modal. - benchmarkCommands: { - speed: -`python3 -m sglang.bench_serving \\ - --backend sglang \\ - --host {{CURL_HOST}} --port {{CURL_PORT}} \\ - --model {{MODEL_NAME}} \\ - --dataset-name {{DATASET}} \\ - --random-input-len {{ISL}} --random-output-len {{OSL}} \\ - --random-range-ratio 1.0 \\ - --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ - --warmup-requests 64 --flush-cache`, - accuracy: { - gsm8k_pct: -`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval -sgl-eval run gsm8k \\ - --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ - --num-threads 32`, - aime25_pct: -`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval -sgl-eval run aime25 \\ - --model {{MODEL_NAME}} --api-key \\ - --n-repeats 16 --max-tokens 64000 \\ - --temperature 1.0 --top-p 0.95 --thinking \\ - --out-dir /sgl-workspace/logs \\ - --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, - }, - numPromptsByConc: { 1: 8, 16: 64, 64: 128, 256: 512, 1024: 2048, 4096: 8192 }, - }, - - // Per-variant accuracy applied to every cell; per-cell `accuracy` overrides. - // Both measured via sgl-eval (thinking mode) on H200. aime25 = pass@1 avg-of-16 - // (n-repeats 16, max-tokens 64000, temp 1.0, top-p 0.95); pass@16 100%, majority@16 93.3%. - defaultAccuracy: { - default: { gsm8k_pct: 98.2, aime25_pct: 87.7 }, - }, - - accuracyLabels: [ - ["aime25_pct", "AIME25", "%"], - ["gsm8k_pct", "GSM8K (1-shot)", "%"], - ], - - dockerImages: { - h200: "lmsysorg/sglang:latest", - b200: "lmsysorg/sglang:latest", - gb300: "lmsysorg/sglang:latest", - b300: "lmsysorg/sglang:latest", - mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618", - mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", - mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", - }, - - github: { - cookbookModel: "zai-org/glm-5.2", - }, - - playgroundFeatures: { - - // ----- Card 1: "Attention Parallelism" ----- - // DSA prefill Context Parallelism (CP) splits the long-prefill attention across - // `cp` ranks — runs on Hopper (H200) and Blackwell (B200/GB300/B300). - // CP sizes auto-gate in the engine to the runtime derivation - // attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden). - // CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention - // currently fails the runtime's dp_size == 1 assert but is allowed here - // with a warning (combined support is planned upstream). - // Strategy knob: interleave (ex round-robin-split) is the layout verified - // here and the default; zigzag (ex in-seq-split) is exposed as an - // experiment — the runtime auto-configures deepep + ep=tp for it and - // restricts it to batch_size=1 (long-context single-request runs). - attention: { - knobs: [ - { id: "tp", label: "TP", values: [null, 4, 8] }, - { id: "cp", label: "CP (DSA prefill)", - values: [null, { value: 1, label: "Off" }, 4, 8], - disable: [ - { when: { hw: ["mi355x", "mi325x", "mi300x"] }, - reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, - { when: { nodes: ["multi-2"] }, - reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, - ] }, - { id: "cpStrategy", label: "CP Strategy", - values: [ - null, - "interleave", - { value: "zigzag", label: "zigzag (experimental)" }, - ], - disable: [ - { when: { hw: ["mi355x", "mi325x", "mi300x"] }, - reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, - { when: { nodes: ["multi-2"] }, - reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, - ] }, - { id: "dpAttn", label: "DP-Attention", - values: [null, false, 4, 8], - labels: { "auto": "Auto", "false": "Off" } }, - ], - }, - - // ----- Card 2: "MoE Parallelism" ----- - moe: { - backend: { - options: [ - { id: null, label: "Inherited" }, - { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, - ], - }, - ep: { label: "EP", values: [null, 4, 8] }, - }, - - // ----- Card 3: "Parsers" ----- - parsers: { - items: [ - { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser glm45" }, - { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser glm47" }, - ], - }, - - // ----- Card 4: "Speculative Decoding" ----- - // GLM-5.2 ships a single MTP (nextn) layer; index_share_for_mtp_iteration reuses the - // DSA indexer topk across draft steps (topk==1 only). - speculative: { - options: [ - { id: "current", label: "Inherited from base" }, - { id: "off", label: "Off (greedy)" }, - { id: "mtp-516", label: "EAGLE / MTP 5-1-6 (low-latency)", - flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 5", - "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6"], - disable: { hw: ["mi355x", "mi325x", "mi300x"] }, - disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, - { id: "mtp-112", label: "EAGLE / MTP 1-1-2 (balanced)", - flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1", - "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"], - disable: { hw: ["mi355x", "mi325x", "mi300x"] }, - disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, - ], - }, - - // ----- Card 5: "PD Disaggregation" ----- - // GLM-5.2 is a DSA model (same family as DeepSeek-V3.2/V4) and supports - // prefill/decode disaggregation. Owns the `--disaggregation-*` flags; the - // engine also pins role-specific serving ports (spaced apart) so prefill + - // decode don't collide on one host. - pdDisagg: { - modes: [ - { id: "off", label: "Off" }, - { id: "prefill", label: "Prefill role" }, - { id: "decode", label: "Decode role" }, - ], - transferBackends: [ - // Mooncake (recommended). The NCCL/MNNVL env is only needed on the - // NVLink-multinode Grace-Blackwell platform (GB300 here). - { id: "mooncake", label: "Mooncake", - env: [ - "NCCL_MNNVL_ENABLE=1", - "NCCL_CUMEM_ENABLE=1", - "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", - "MC_FORCE_MNNVL=1", - ], - envWhen: { hw: ["gb300"] } }, - { id: "nixl", label: "NiXL" }, - ], - // No IB-device knob: mooncake auto-detects the HCA. Pass - // --disaggregation-ib-device only if discovery picks the wrong NIC - // (see Configuration Tips). - // Router fronting the prefill + decode roles; substitute /. - router: { - port: 8000, - command: -`python3 -m sglang_router.launch_router \\ - --pd-disaggregation \\ - --prefill http://:{{PREFILL_PORT}} \\ - --decode http://:{{DECODE_PORT}} \\ - --host 0.0.0.0 --port {{ROUTER_PORT}} \\ - --disable-circuit-breaker \\ - --health-check-interval-secs 999999`, - }, - }, - - // ----- Card 6: "Hierarchical KV Cache" ----- - hicache: { - backends: [ - { id: null, label: "Auto" }, - { id: "file", label: "File" }, - { id: "mooncake", label: "Mooncake" }, - ], - writePolicies: [ - { id: "auto", label: "Auto" }, - { id: "write_through", label: "Write-through" }, - { id: "write_back", label: "Write-back" }, - ], - }, - }, - - cells: [ - // ==================================================================== - // H200 + FP8 (Hopper) — TP8. CP (DSA prefill) verified here. - // ==================================================================== - { - match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.8", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - // Large chunked-prefill is the dominant balanced lever (prefill-bound at this - // concurrency); max-running tracks KV capacity (~60-80 for 8K+1K reqs on 8xH200). - "--chunked-prefill-size 32768", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // B200 + FP8 (Blackwell) — TP8. low-latency verified on b200-verda-k8s - // ==================================================================== - { - match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.8", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - // Large chunked-prefill is the dominant balanced lever (prefill-bound at this - // concurrency); max-running tracks KV capacity (~89 for 8K+1K reqs on 8xB200). - "--chunked-prefill-size 32768", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // GB300 + FP8 (Grace-Blackwell, 4-GPU single node) — TP4. - // Flags mirror the B200 (sm100) configs; all three strategies verified end-to-end on - // a single 4xGB300 node (v0.5.13.post1). GB300 leads B200 per-GPU in every regime. - // Stage the weights on node-local NVMe first — shared cluster-storage reads are slow. - // ==================================================================== - { - match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--dp 4", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - // Same prefill lever as H200/B200 balanced; max-running tracks the TP4 KV capacity. - "--chunked-prefill-size 32768", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [ - "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512", - ], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--dp 4", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // B300 + FP8 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1). - // Recipe mirrors the verified B200 (sm100) FP8 path. B300 (sm103) currently trails B200 per-GPU - // because deep_gemm/DSA are tuned for sm100; expected to improve as sm103 gets first-class kernels. - // ==================================================================== - { - match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.8", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - "--chunked-prefill-size 32768", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--moe-a2a-backend deepep", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // B300 + BF16 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1). - // The unquantized GLM-5.2 (~700B, ~1.51 TB) only fits single-node on 8xB300 - // (~2.1 TB HBM); smaller GPUs need multi-node (e.g. 2x 8xH200). balanced/HT run plain TP8 - // (no DP-Attention/DeepEP), so they trail the FP8 recipe at high concurrency. - // ==================================================================== - { - match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.9", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.9", - "--chunked-prefill-size 32768", - "--max-running-requests 80", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--mem-fraction-static 0.9", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // BF16 multi-node (inferred) — the 1.51 TB checkpoint spread over 2 nodes. - // 2x 8xH200 / 2x 8xB200 at TP16, 2x 4xGB300 at TP8. The engine injects - // --nnodes / --node-rank / --dist-init-addr from the Multi-Nodes selector. - // Recipes inferred from the single-node B300 path; not benchmarked → verified:false. - // ==================================================================== - { - match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - "--chunked-prefill-size 32768", - "--max-running-requests 80", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - "--chunked-prefill-size 32768", - "--max-running-requests 80", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 16", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 1", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 2", - "--mem-fraction-static 0.85", - "--chunked-prefill-size 32768", - "--max-running-requests 80", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--mem-fraction-static 0.85", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - - // ==================================================================== - // NVFP4 — nvidia/GLM-5.2-NVFP4 (Model Optimizer). TP8 on B200/B300, TP4 on GB300. - // B200/B300: 8-GPU single node, TP8 (low-latency / balanced / high-throughput); balanced & - // high-throughput add DP-Attention (dp8). low-latency uses MTP 5-1-6, balanced MTP 2-1-3. - // GB300: 4-GPU single node, TP4 (the node fits the ~381 GB build); GB300 adds dp4 on - // balanced & high-throughput; low-latency uses MTP 5-1-6. - // ==================================================================== - { - match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--quantization modelopt_fp4", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--quantization modelopt_fp4", - "--dp 8", - "--enable-dp-attention", - // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the - // verify overhead of a long draft outweighs the accept-length gain. - "--speculative-algorithm EAGLE", - "--speculative-num-steps 2", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 3", - // Larger chunked-prefill (32768 → ~4096/rank under dp8) is the dominant balanced lever. - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.92", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--quantization modelopt_fp4", - "--dp 8", - "--enable-dp-attention", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.92", - "--max-running-requests 512", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--quantization modelopt_fp4", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.85", - "--max-running-requests 16", - "--cuda-graph-max-bs 16", - "--max-prefill-tokens 8192", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--quantization modelopt_fp4", - // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the - // verify overhead of a long draft outweighs the accept-length gain. - "--speculative-algorithm EAGLE", - "--speculative-num-steps 2", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 3", - // Two required flags for DP-Attention + MTP here: `decode`-mode spec attention - // avoids a CUDA-graph capture deadlock, and max-running 256 lifts the default - // ~48-request throttle so DP-Attention can fill all 8 ranks. - "--speculative-attention-mode decode", - "--max-running-requests 256", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dp 8", - "--enable-dp-attention", - "--quantization modelopt_fp4", - "--max-running-requests 1024", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.85", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--quantization modelopt_fp4", - "--speculative-algorithm EAGLE", - "--speculative-num-steps 5", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 6", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.85", - "--max-running-requests 16", - "--cuda-graph-max-bs 16", - "--max-prefill-tokens 8192", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--quantization modelopt_fp4", - "--dp 4", - "--enable-dp-attention", - // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the - // verify overhead of a long draft outweighs the accept-length gain. - "--speculative-algorithm EAGLE", - "--speculative-num-steps 2", - "--speculative-eagle-topk 1", - "--speculative-num-draft-tokens 3", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.92", - "--max-running-requests 256", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 4", - "--quantization modelopt_fp4", - "--dp 4", - "--enable-dp-attention", - "--chunked-prefill-size 8192", - "--mem-fraction-static 0.92", - "--max-running-requests 512", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - // ==================================================================== - // AMD MI300X / MI325X / MI355X (ROCm) — TP8, DSA tilelang backend. - // No MTP: disabled in the Speculative card for AMD (the gfx950 spec-decode - // draft kernel is not yet validated, and num-steps>3 hits a separate build - // issue). Strategies differ only by batch-shaping levers - // (cuda-graph-max-bs / max-running-requests / chunked-prefill): - // low-latency — large chunked-prefill, default bs. - // balanced — chunked-prefill 32768 + bs128, max-running 80. - // high-throughput — bs256, max-running 256. - // ACCURACY: the earlier gfx950 block-FP8 bpreshuffle miscompile (GSM8K ~0) is - // fixed as of the pinned mi355x image (...-20260618); MI355X FP8 was re-validated - // (GSM8K ~0.96, NIAH 15/15 to ~118K) and all three FP8 strategies are benchmarked - // + marked verified:true (see glm-5.2-benchmarks.jsx). All BF16 and all gfx942 - // (MI325X/MI300X) cells stay verified:false (not yet benchmarked, but correct). - // BF16 (~1.51 TB) only fits single-node on MI325X (2 TB) / MI355X (2.3 TB); - // MI300X (1.5 TB) needs multi-node, so its BF16 cells are omitted. - // ==================================================================== - { - match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 131072", - "--mem-fraction-static 0.80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 128", - "--max-running-requests 80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: true, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 256", - "--max-running-requests 256", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 131072", - "--mem-fraction-static 0.80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 128", - "--max-running-requests 80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 256", - "--max-running-requests 256", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 131072", - "--mem-fraction-static 0.80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 128", - "--max-running-requests 80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 256", - "--max-running-requests 256", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 131072", - "--mem-fraction-static 0.80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 128", - "--max-running-requests 80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 256", - "--max-running-requests 256", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 131072", - "--mem-fraction-static 0.80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--chunked-prefill-size 32768", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 128", - "--max-running-requests 80", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, - verified: false, - env: [], - flags: [ - "--model-path {{MODEL_NAME}}", - "--tp 8", - "--dsa-prefill-backend tilelang", - "--dsa-decode-backend tilelang", - "--mem-fraction-static 0.85", - "--cuda-graph-max-bs 256", - "--max-running-requests 256", - "--watchdog-timeout 1200", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - ], -}; +// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration). +// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr`/`--host`/`--port` literals — engine injects them. + +export const config = { + modelName: "GLM-5.2", + + supportedHardware: [ + "h200", "b200", "gb300", "b300", + "mi355x", "mi325x", "mi300x", + ], + + // Single released checkpoint — no size/mode split. + variants: [ + { id: "default", label: "GLM-5.2", subtitle: "MoE · DSA" }, + ], + quantizations: [ + { id: "fp8", label: "FP8" }, + { id: "bf16", label: "BF16" }, + { id: "nvfp4", label: "NVFP4" }, + { id: "mxfp4", label: "MXFP4" }, + ], + strategies: [ + { id: "low-latency", label: "Low-Latency" }, + { id: "balanced", label: "Balanced" }, + { id: "high-throughput", label: "High-Throughput" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + { id: "multi-2", label: "Multi-Nodes" }, + ], + + modelNames: { + "default|fp8": "zai-org/GLM-5.2-FP8", + "default|bf16": "zai-org/GLM-5.2", + "default|nvfp4": "nvidia/GLM-5.2-NVFP4", + "default|mxfp4": "amd/GLM-5.2-MXFP4", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + // Reproduce commands for the Benchmark card's "⚡ Reproduce" modal. + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --random-range-ratio 1.0 \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --warmup-requests 64 --flush-cache`, + accuracy: { + gsm8k_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --num-threads 32`, + aime25_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run aime25 \\ + --model {{MODEL_NAME}} --api-key \\ + --n-repeats 16 --max-tokens 64000 \\ + --temperature 1.0 --top-p 0.95 --thinking \\ + --out-dir /sgl-workspace/logs \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, + }, + numPromptsByConc: { 1: 8, 16: 64, 64: 128, 256: 512, 1024: 2048, 4096: 8192 }, + }, + + // Per-variant accuracy applied to every cell; per-cell `accuracy` overrides. + // Both measured via sgl-eval (thinking mode) on H200. aime25 = pass@1 avg-of-16 + // (n-repeats 16, max-tokens 64000, temp 1.0, top-p 0.95); pass@16 100%, majority@16 93.3%. + defaultAccuracy: { + default: { gsm8k_pct: 98.2, aime25_pct: 87.7 }, + }, + + accuracyLabels: [ + ["aime25_pct", "AIME25", "%"], + ["gsm8k_pct", "GSM8K (1-shot)", "%"], + ], + + dockerImages: { + h200: "lmsysorg/sglang:latest", + b200: "lmsysorg/sglang:latest", + gb300: "lmsysorg/sglang:latest", + b300: "lmsysorg/sglang:latest", + mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618", + "mi355x|mxfp4": "lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728", + mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", + mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", + }, + + github: { + cookbookModel: "zai-org/glm-5.2", + }, + + playgroundFeatures: { + + // ----- Card 1: "Attention Parallelism" ----- + // DSA prefill Context Parallelism (CP) splits the long-prefill attention across + // `cp` ranks — runs on Hopper (H200) and Blackwell (B200/GB300/B300). + // CP sizes auto-gate in the engine to the runtime derivation + // attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden). + // CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention + // currently fails the runtime's dp_size == 1 assert but is allowed here + // with a warning (combined support is planned upstream). + // Strategy knob: interleave (ex round-robin-split) is the layout verified + // here and the default; zigzag (ex in-seq-split) is exposed as an + // experiment — the runtime auto-configures deepep + ep=tp for it and + // restricts it to batch_size=1 (long-context single-request runs). + attention: { + knobs: [ + { id: "tp", label: "TP", values: [null, 4, 8] }, + { id: "cp", label: "CP (DSA prefill)", + values: [null, { value: 1, label: "Off" }, 4, 8], + disable: [ + { when: { hw: ["mi355x", "mi325x", "mi300x"] }, + reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, + { when: { nodes: ["multi-2"] }, + reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, + ] }, + { id: "cpStrategy", label: "CP Strategy", + values: [ + null, + "interleave", + { value: "zigzag", label: "zigzag (experimental)" }, + ], + disable: [ + { when: { hw: ["mi355x", "mi325x", "mi300x"] }, + reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, + { when: { nodes: ["multi-2"] }, + reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, + ] }, + { id: "dpAttn", label: "DP-Attention", + values: [null, false, 4, 8], + labels: { "auto": "Auto", "false": "Off" } }, + ], + }, + + // ----- Card 2: "MoE Parallelism" ----- + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, + ], + }, + ep: { label: "EP", values: [null, 4, 8] }, + }, + + // ----- Card 3: "Parsers" ----- + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser glm45" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser glm47" }, + ], + }, + + // ----- Card 4: "Speculative Decoding" ----- + // GLM-5.2 ships a single MTP (nextn) layer; index_share_for_mtp_iteration reuses the + // DSA indexer topk across draft steps (topk==1 only). + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "mtp-516", label: "EAGLE / MTP 5-1-6 (low-latency)", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 5", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6"], + disable: { hw: ["mi355x", "mi325x", "mi300x"] }, + disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, + { id: "mtp-112", label: "EAGLE / MTP 1-1-2 (balanced)", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"], + disable: { hw: ["mi355x", "mi325x", "mi300x"] }, + disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, + { id: "mtp-314", label: "EAGLE / MTP 3-1-4 (agentic · MI355X MXFP4)", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"], + enable: { hw: ["mi355x"], quant: ["mxfp4"] }, + enableReason: "Validated on MI355X gfx950 with amd/GLM-5.2-MXFP4 (InferenceX AgentX sweep, GSM8K em_strict 0.971). num-steps=3 stays within the validated gfx950 spec-decode build envelope (≤3). Pair with SGLANG_SIMULATE_ACC_LEN=2.99 for benchmarking (golden AL from golden_al_distribution/glm5.2_mtp.yaml, thinking_on, num_speculative_tokens=3)." }, + ], + }, + + // ----- Card 5: "PD Disaggregation" ----- + // GLM-5.2 is a DSA model (same family as DeepSeek-V3.2/V4) and supports + // prefill/decode disaggregation. Owns the `--disaggregation-*` flags; the + // engine also pins role-specific serving ports (spaced apart) so prefill + + // decode don't collide on one host. + pdDisagg: { + modes: [ + { id: "off", label: "Off" }, + { id: "prefill", label: "Prefill role" }, + { id: "decode", label: "Decode role" }, + ], + transferBackends: [ + // Mooncake (recommended). The NCCL/MNNVL env is only needed on the + // NVLink-multinode Grace-Blackwell platform (GB300 here). + { id: "mooncake", label: "Mooncake", + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", + "MC_FORCE_MNNVL=1", + ], + envWhen: { hw: ["gb300"] } }, + { id: "nixl", label: "NiXL" }, + ], + // No IB-device knob: mooncake auto-detects the HCA. Pass + // --disaggregation-ib-device only if discovery picks the wrong NIC + // (see Configuration Tips). + // Router fronting the prefill + decode roles; substitute /. + router: { + port: 8000, + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} \\ + --decode http://:{{DECODE_PORT}} \\ + --host 0.0.0.0 --port {{ROUTER_PORT}} \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`, + }, + }, + + // ----- Card 6: "Hierarchical KV Cache" ----- + hicache: { + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + ], + }, + }, + + cells: [ + // ==================================================================== + // H200 + FP8 (Hopper) — TP8. CP (DSA prefill) verified here. + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Large chunked-prefill is the dominant balanced lever (prefill-bound at this + // concurrency); max-running tracks KV capacity (~60-80 for 8K+1K reqs on 8xH200). + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B200 + FP8 (Blackwell) — TP8. low-latency verified on b200-verda-k8s + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Large chunked-prefill is the dominant balanced lever (prefill-bound at this + // concurrency); max-running tracks KV capacity (~89 for 8K+1K reqs on 8xB200). + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 + FP8 (Grace-Blackwell, 4-GPU single node) — TP4. + // Flags mirror the B200 (sm100) configs; all three strategies verified end-to-end on + // a single 4xGB300 node (v0.5.13.post1). GB300 leads B200 per-GPU in every regime. + // Stage the weights on node-local NVMe first — shared cluster-storage reads are slow. + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Same prefill lever as H200/B200 balanced; max-running tracks the TP4 KV capacity. + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [ + "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 + FP8 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1). + // Recipe mirrors the verified B200 (sm100) FP8 path. B300 (sm103) currently trails B200 per-GPU + // because deep_gemm/DSA are tuned for sm100; expected to improve as sm103 gets first-class kernels. + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 + BF16 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1). + // The unquantized GLM-5.2 (~700B, ~1.51 TB) only fits single-node on 8xB300 + // (~2.1 TB HBM); smaller GPUs need multi-node (e.g. 2x 8xH200). balanced/HT run plain TP8 + // (no DP-Attention/DeepEP), so they trail the FP8 recipe at high concurrency. + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.9", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.9", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--mem-fraction-static 0.9", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // BF16 multi-node (inferred) — the 1.51 TB checkpoint spread over 2 nodes. + // 2x 8xH200 / 2x 8xB200 at TP16, 2x 4xGB300 at TP8. The engine injects + // --nnodes / --node-rank / --dist-init-addr from the Multi-Nodes selector. + // Recipes inferred from the single-node B300 path; not benchmarked → verified:false. + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // NVFP4 — nvidia/GLM-5.2-NVFP4 (Model Optimizer). TP8 on B200/B300, TP4 on GB300. + // B200/B300: 8-GPU single node, TP8 (low-latency / balanced / high-throughput); balanced & + // high-throughput add DP-Attention (dp8). low-latency uses MTP 5-1-6, balanced MTP 2-1-3. + // GB300: 4-GPU single node, TP4 (the node fits the ~381 GB build); GB300 adds dp4 on + // balanced & high-throughput; low-latency uses MTP 5-1-6. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--dp 8", + "--enable-dp-attention", + // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the + // verify overhead of a long draft outweighs the accept-length gain. + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + // Larger chunked-prefill (32768 → ~4096/rank under dp8) is the dominant balanced lever. + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.92", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--dp 8", + "--enable-dp-attention", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.92", + "--max-running-requests 512", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--max-running-requests 16", + "--cuda-graph-max-bs 16", + "--max-prefill-tokens 8192", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--quantization modelopt_fp4", + // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the + // verify overhead of a long draft outweighs the accept-length gain. + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + // Two required flags for DP-Attention + MTP here: `decode`-mode spec attention + // avoids a CUDA-graph capture deadlock, and max-running 256 lifts the default + // ~48-request throttle so DP-Attention can fill all 8 ranks. + "--speculative-attention-mode decode", + "--max-running-requests 256", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--quantization modelopt_fp4", + "--max-running-requests 1024", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--max-running-requests 16", + "--cuda-graph-max-bs 16", + "--max-prefill-tokens 8192", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--dp 4", + "--enable-dp-attention", + // Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the + // verify overhead of a long draft outweighs the accept-length gain. + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.92", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--dp 4", + "--enable-dp-attention", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.92", + "--max-running-requests 512", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ==================================================================== + // AMD MI300X / MI325X / MI355X (ROCm) — TP8, DSA tilelang backend. + // No MTP: disabled in the Speculative card for AMD (the gfx950 spec-decode + // draft kernel is not yet validated, and num-steps>3 hits a separate build + // issue). Strategies differ only by batch-shaping levers + // (cuda-graph-max-bs / max-running-requests / chunked-prefill): + // low-latency — large chunked-prefill, default bs. + // balanced — chunked-prefill 32768 + bs128, max-running 80. + // high-throughput — bs256, max-running 256. + // ACCURACY: the earlier gfx950 block-FP8 bpreshuffle miscompile (GSM8K ~0) is + // fixed as of the pinned mi355x image (...-20260618); MI355X FP8 was re-validated + // (GSM8K ~0.96, NIAH 15/15 to ~118K) and all three FP8 strategies are benchmarked + // + marked verified:true (see glm-5.2-benchmarks.jsx). All BF16 and all gfx942 + // (MI325X/MI300X) cells stay verified:false (not yet benchmarked, but correct). + // BF16 (~1.51 TB) only fits single-node on MI325X (2 TB) / MI355X (2.3 TB); + // MI300X (1.5 TB) needs multi-node, so its BF16 cells are omitted. + // ==================================================================== + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ==================================================================== + // AMD MI355X + MXFP4 — amd/GLM-5.2-MXFP4 (Quark). TP4: the 4-bit MoE + // weights fit a 4-GPU slice, mirroring the amd/GLM-5.1-MXFP4 MI355X recipe (same DSA + // architecture family) — --trust-remote-code (Quark custom quant config) + // and --kv-cache-dtype fp8_e4m3 both come from that precedent. Pinned to a + // newer image (v0.5.16, see dockerImages["mi355x|mxfp4"]) than the FP8/BF16 + // mi355x cells. MTP (mtp-314, steps=3) is validated on MI355X gfx950 with + // this precision — see the mtp-314 cell below. Not yet benchmarked for + // GLM-5.2 on the base strategies → verified:false. + // ==================================================================== + { + match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--kv-cache-dtype fp8_e4m3", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--kv-cache-dtype fp8_e4m3", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--kv-cache-dtype fp8_e4m3", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // MI355X + MXFP4 + MTP (mtp-314): validated AgentX recipe. + // steps=3 stays within the gfx950 spec-decode build envelope (≤3). + // mem-fraction-static 0.80: headroom for MTP draft buffer on top of + // 4-bit MoE weights + KV cache (matches InferenceX AgentX harness conc≤16). + // For benchmarking: set SGLANG_SIMULATE_ACC_LEN=2.99, + // SGLANG_SIMULATE_ACC_METHOD=match-expected, + // SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token + // (golden AL: golden_al_distribution/glm5.2_mtp.yaml, thinking_on, num_speculative_tokens=3). + { + match: { hw: "mi355x", variant: "default", quant: "mxfp4", strategy: "mtp-314", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--kv-cache-dtype fp8_e4m3", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 3", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 4", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--cuda-graph-max-bs 160", + "--max-running-requests 160", + "--watchdog-timeout 1800", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};