diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx new file mode 100644 index 000000000..640414a2f --- /dev/null +++ b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx @@ -0,0 +1,259 @@ +--- +title: GLM-5.3 +description: "Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash Command +pip install --upgrade pip +pip install uv +uv pip install --prerelease=allow sglang +``` + +Then run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +docker pull lmsysorg/sglang:latest +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: + +- **Low-Latency** — fastest reply for a single user. Pick for chat. +- **Balanced** — good speed with several users at once. Use for typical multi-user serving. +- **High-Throughput** — most tokens per second across many users. Best for batch jobs. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/zai-org/glm-5.3.jsx"; +import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx"; + + + + + All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.3. + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the deployment matrix**. It lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +**GLM-5.3** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It keeps the same base architecture while updating post-training for complex coding and long-horizon tasks. It ships in two precisions — **FP8** (`zai-org/GLM-5.3`) and full **BF16** (`zai-org/GLM-5.3-BF16`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, RadixArk publishes an **experimental NVFP4** build (`RadixArk/GLM-5.3-NVFP4`, Model Optimizer) that quantizes only the routed experts' linear weights and activations to 4-bit (attention, shared experts, dense layers, MTP, embeddings, and the LM head stay unquantized), cutting the weight footprint to ~0.45 TB so a 4-GPU GB300 node can serve it with TP4. **The NVFP4 cells are experimental** — benchmark data (accuracy / throughput) is still pending; treat them as provisional until the benchmark cards fill in. + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelArchitectureContext
GLM-5.3MoE · DSA · 256 experts (top-8) · MTP · FP81,048,576
GLM-5.3-BF16MoE · DSA · 256 experts (top-8) · MTP · BF161,048,576
GLM-5.3-NVFP4MoE · DSA · 256 experts (top-8) · MTP · NVFP41,048,576
+ +**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). + +**Resources:** [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) · [GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16) · [GLM-5.3-NVFP4](https://huggingface.co/RadixArk/GLM-5.3-NVFP4). + +## 2. Configuration Tips + +- **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. +- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work. +- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). +- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. +- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. +- **PD Disaggregation (prefill/decode).** GLM-5.3 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. +- **Chunked-prefill size is regime-dependent.** For long-input balanced workloads, start with `--chunked-prefill-size 32768` and tune it together with `--max-running-requests` for your input length and KV capacity. Keep the default chunked-prefill size for the high-throughput recipe unless profiling shows that prefill is the bottleneck. + +- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.3`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.3-BF16`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there. Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). GLM-5.3 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). +- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel because the gfx950 spec-decode draft kernel is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP. + +## 3. Advanced Usage + +### 3.1 Reasoning + +GLM-5.3 is a reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response. + +**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to select `low`, `high`, or `max`. If you omit it or pass another value, the template uses `max`. + +| `reasoning_effort` | Injected system line | Effect | +|---|---|---| +| *(not passed / unset)*, `"max"` | `Reasoning Effort: Max` | default — highest reasoning effort | +| `"high"` | `Reasoning Effort: High` | high reasoning effort | +| `"low"` | `Reasoning Effort: Low` | low reasoning effort | +| any other value | `Reasoning Effort: Max` | falls back to the default | + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="zai-org/GLM-5.3", + messages=[{"role": "user", "content": "What is 15% of 240?"}], + extra_body={"chat_template_kwargs": {"clear_thinking": True, "reasoning_effort": "high"}}, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) +``` + + + + + +```text Output +Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240. +2. **Convert the percentage to a decimal:** 15% = 0.15 +3. **Multiply by the total:** 0.15 * 240 = 36 + (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.) + +Answer: 15% of 240 is **36**. + +Here is how you can calculate it: +0.15 × 240 = 36 +``` + + + +### 3.2 Tool Calling + +Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.3 emits the newer `………` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, +}] +resp = client.chat.completions.create( + model="zai-org/GLM-5.3", + messages=[{"role": "user", "content": "What's the weather in Paris?"}], + tools=tools, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Tool calls:", msg.tool_calls) +``` + + + + + +```text Output +Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city. + +Tool calls: [ + { + "id": "call_13fcd52146934b7781d06d4a", + "type": "function", + "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"} + } +] +``` + + + +### 3.3 HiCache (Hierarchical KV Caching) + +For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.3's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. + +### 3.4 Claude Code Integration + +GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: + +```bash Command +export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" +export ANTHROPIC_AUTH_TOKEN="dummy" +export API_TIMEOUT_MS="3000000" +export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" +export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 +export CLAUDE_CODE_ATTRIBUTION_HEADER=0 +export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.3[1m]" +export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.3[1m]" +export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.3[1m]" +claude +``` + +Two of these matter specifically for GLM-5.3: + +- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.3's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. +- **`glm-5.3[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.3's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. + +For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). + +### 3.5 Context Parallelism + +Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments: +```bash +--attn-cp-size 8 \ +--enable-prefill-cp \ +--cp-strategy interleave \ +``` +which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill. + +When deploying with PD Disaggregation, a **prefill worker using the Mooncake transfer backend** can enable the [LayerSplit](https://z.ai/blog/scaling-pain) technique with +```bash +--enable-dsa-cache-layer-split \ +--enable-prefill-cp \ +--attn-cp-size 8 \ +--cp-strategy interleave \ +``` +With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. diff --git a/docs/docs.json b/docs/docs.json index bea6274d9..9dc1a9c1d 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1226,6 +1226,7 @@ { "group": "GLM", "pages": [ + "cookbook/autoregressive/GLM/GLM-5.3", "cookbook/autoregressive/GLM/GLM-5.3-Flash", "cookbook/autoregressive/GLM/GLM-5.2", "cookbook/autoregressive/GLM/GLM-5.1", diff --git a/docs/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx b/docs/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx new file mode 100644 index 000000000..624569077 --- /dev/null +++ b/docs/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx @@ -0,0 +1,221 @@ +// GLM-5.3 benchmark placeholders, keyed by the same `match` tuple as glm-5.3.jsx cells. +// Bare match stubs render as pending until speed and accuracy measurements are available. +export const benchmarks = [ + // NVIDIA FP8, single node. + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.42 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 780, tpot_ms: 3.71, tokens_per_sec_per_gpu: 252 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 5499, tpot_ms: 14.20, tokens_per_sec_per_gpu: 920 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.496 at concurrency 1, 3.476 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.12, aime25_pct: 91.88 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 7790, tpot_ms: 21.90, tokens_per_sec_per_gpu: 2440 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 57607, tpot_ms: 33.39, tokens_per_sec_per_gpu: 2558 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.12 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 400, tpot_ms: 3.11, tokens_per_sec_per_gpu: 321 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 3044, tpot_ms: 7.84, tokens_per_sec_per_gpu: 1662 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.507 at concurrency 1, 3.503 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.42, aime25_pct: 90.83 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 5908, tpot_ms: 16.53, tokens_per_sec_per_gpu: 3220 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 18916, tpot_ms: 31.88, tokens_per_sec_per_gpu: 5025 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.73 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 372, tpot_ms: 3.68, tokens_per_sec_per_gpu: 556 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 3305, tpot_ms: 10.02, tokens_per_sec_per_gpu: 2718 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.489 at concurrency 1, 3.509 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.19, aime25_pct: 90.62 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 7775, tpot_ms: 22.77, tokens_per_sec_per_gpu: 4743 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 25775, tpot_ms: 45.87, tokens_per_sec_per_gpu: 7501 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.12 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 342, tpot_ms: 3.10, tokens_per_sec_per_gpu: 328 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 2851, tpot_ms: 7.77, tokens_per_sec_per_gpu: 1705 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.485 at concurrency 1, 3.487 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.42, aime25_pct: 92.08 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 5586, tpot_ms: 16.37, tokens_per_sec_per_gpu: 3296 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 17848, tpot_ms: 32.48, tokens_per_sec_per_gpu: 5259 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + + // NVIDIA NVFP4, single node (RadixArk/GLM-5.3-NVFP4). + { match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" } }, + { match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" } }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 26fd7fdaa273", + accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 281, tpot_ms: 1.48, tokens_per_sec_per_gpu: 640 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 1988, tpot_ms: 5.11, tokens_per_sec_per_gpu: 2306 }, + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 26fd7fdaa273", + accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 5227, tpot_ms: 12.40, tokens_per_sec_per_gpu: 3748 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 17513, tpot_ms: 29.97, tokens_per_sec_per_gpu: 5416 }, + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 26fd7fdaa273", + accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 307, tpot_ms: 1.71, tokens_per_sec_per_gpu: 1113 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 2132, tpot_ms: 6.18, tokens_per_sec_per_gpu: 3979 }, + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 26fd7fdaa273", + accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 5828, tpot_ms: 16.69, tokens_per_sec_per_gpu: 6108 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 22330, tpot_ms: 39.63, tokens_per_sec_per_gpu: 8537 }, + ], + }, + + // NVIDIA BF16. + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.12 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 }, + ttft_ms: 344, tpot_ms: 2.96, tokens_per_sec_per_gpu: 341 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 }, + ttft_ms: 2772, tpot_ms: 9.13, tokens_per_sec_per_gpu: 1521 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.531 at concurrency 1, 3.506 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.27, aime25_pct: 93.75 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 }, + ttft_ms: 9512, tpot_ms: 22.30, tokens_per_sec_per_gpu: 2279 }, + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 }, + ttft_ms: 75250, tpot_ms: 25.84, tokens_per_sec_per_gpu: 2333 }, + ], + notes: + "Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.", + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + sglang_version: "main @ 20a491d1d311", + accuracy: { gsm8k_pct: 97.12 }, + speed: [ + { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 }, + ttft_ms: 349227, tpot_ms: 65.45, tokens_per_sec_per_gpu: 2216 }, + ], + }, + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } }, + { match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } }, + { match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } }, + { match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } }, + + // AMD ROCm, single node. + { match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } }, + { match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } }, + { match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } }, + { match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } }, + { match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } }, + { match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } }, + { match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } }, +]; diff --git a/docs/src/snippets/configs/zai-org/glm-5.3.jsx b/docs/src/snippets/configs/zai-org/glm-5.3.jsx new file mode 100644 index 000000000..31c6a8f40 --- /dev/null +++ b/docs/src/snippets/configs/zai-org/glm-5.3.jsx @@ -0,0 +1,1028 @@ +// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration). +// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr`/`--host`/`--port` literals — engine injects them. + +export const config = { + modelName: "GLM-5.3", + + supportedHardware: [ + "h200", "b200", "gb300", "b300", + "mi355x", "mi325x", "mi300x", + ], + + // Single released checkpoint — no size/mode split. + variants: [ + { id: "default", label: "GLM-5.3", subtitle: "MoE · DSA" }, + ], + quantizations: [ + { id: "fp8", label: "FP8" }, + { id: "bf16", label: "BF16" }, + { id: "nvfp4", label: "NVFP4 (Experimental)" }, + ], + strategies: [ + { id: "low-latency", label: "Low-Latency" }, + { id: "balanced", label: "Balanced" }, + { id: "high-throughput", label: "High-Throughput" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + { id: "multi-2", label: "Multi-Nodes" }, + ], + + modelNames: { + "default|fp8": "zai-org/GLM-5.3", + "default|bf16": "zai-org/GLM-5.3-BF16", + "default|nvfp4": "RadixArk/GLM-5.3-NVFP4", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + // Reproduce commands for the Benchmark card's "⚡ Reproduce" modal. + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} \\ + --random-range-ratio 1.0 \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --warmup-requests 64 --flush-cache`, + accuracy: { + gsm8k_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run gsm8k \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ + --num-threads 32`, + aime26_pct: +`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +sgl-eval run aime26 \\ + --model {{MODEL_NAME}} --api-key \\ + --n-repeats 16 --max-tokens 64000 \\ + --temperature 1.0 --top-p 0.95 --thinking \\ + --out-dir /sgl-workspace/logs \\ + --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1`, + }, + numPromptsByConc: { 1: 8, 16: 64, 64: 128, 256: 512, 1024: 2048, 4096: 8192 }, + }, + + accuracyLabels: [ + ["aime26_pct", "AIME26", "%"], + ["gsm8k_pct", "GSM8K (1-shot)", "%"], + ], + + dockerImages: { + h200: "lmsysorg/sglang:latest", + b200: "lmsysorg/sglang:latest", + gb300: "lmsysorg/sglang:latest", + b300: "lmsysorg/sglang:latest", + mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260618", + mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", + mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616", + }, + + github: { + cookbookModel: "zai-org/glm-5.3", + }, + + playgroundFeatures: { + + // ----- Card 1: "Attention Parallelism" ----- + // DSA prefill Context Parallelism (CP) splits the long-prefill attention across + // `cp` ranks — runs on Hopper (H200) and Blackwell (B200/GB300/B300). + // CP sizes auto-gate in the engine to the runtime derivation + // attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden). + // CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention + // currently fails the runtime's dp_size == 1 assert but is allowed here + // with a warning (combined support is planned upstream). + // Strategy knob: interleave (ex round-robin-split) is the default; + // zigzag (ex in-seq-split) is exposed as an + // experiment — the runtime auto-configures deepep + ep=tp for it and + // restricts it to batch_size=1 (long-context single-request runs). + attention: { + knobs: [ + { id: "tp", label: "TP", values: [null, 4, 8] }, + { id: "cp", label: "CP (DSA prefill)", + values: [null, { value: 1, label: "Off" }, 4, 8], + disable: [ + { when: { hw: ["mi355x", "mi325x", "mi300x"] }, + reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, + { when: { nodes: ["multi-2"] }, + reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, + ] }, + { id: "cpStrategy", label: "CP Strategy", + values: [ + null, + "interleave", + { value: "zigzag", label: "zigzag (experimental)" }, + ], + disable: [ + { when: { hw: ["mi355x", "mi325x", "mi300x"] }, + reason: "The ROCm DSA-CP path is not yet validated on AMD (MI300X/MI325X/MI355X) — keep CP off there for now." }, + { when: { nodes: ["multi-2"] }, + reason: "Prefill Context Parallel is single-machine only (SGLang asserts tp_size <= 8; cross-machine CP has precision issues)." }, + ] }, + { id: "dpAttn", label: "DP-Attention", + values: [null, false, 4, 8], + labels: { "auto": "Auto", "false": "Off" } }, + ], + }, + + // ----- Card 2: "MoE Parallelism" ----- + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, + ], + }, + ep: { label: "EP", values: [null, 4, 8] }, + }, + + // ----- Card 3: "Parsers" ----- + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser glm45" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser glm47" }, + ], + }, + + // ----- Card 4: "Speculative Decoding" ----- + // GLM-5.3 ships a single MTP (nextn) layer; index_share_for_mtp_iteration reuses the + // DSA indexer topk across draft steps (topk==1 only). + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "mtp-516", label: "EAGLE / MTP 5-1-6 (low-latency)", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 5", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6"], + disable: { hw: ["mi355x", "mi325x", "mi300x"] }, + disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, + { id: "mtp-112", label: "EAGLE / MTP 1-1-2 (balanced)", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 1", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 2"], + disable: { hw: ["mi355x", "mi325x", "mi300x"] }, + disableReason: "MTP/EAGLE speculative decoding is not yet validated on AMD ROCm (MI300X/MI325X/MI355X): the gfx950 spec-decode draft kernel is not yet validated and at --speculative-num-steps > 3 hits a separate build issue; the DSA nextn draft path is CUDA-only." }, + ], + }, + + // ----- Card 5: "PD Disaggregation" ----- + // GLM-5.3 is a DSA model (same family as DeepSeek-V3.2/V4) and supports + // prefill/decode disaggregation. Owns the `--disaggregation-*` flags; the + // engine also pins role-specific serving ports (spaced apart) so prefill + + // decode don't collide on one host. + pdDisagg: { + modes: [ + { id: "off", label: "Off" }, + { id: "prefill", label: "Prefill role" }, + { id: "decode", label: "Decode role" }, + ], + transferBackends: [ + // Mooncake (recommended). The NCCL/MNNVL env is only needed on the + // NVLink-multinode Grace-Blackwell platform (GB300 here). + { id: "mooncake", label: "Mooncake", + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", + "MC_FORCE_MNNVL=1", + ], + envWhen: { hw: ["gb300"] } }, + { id: "nixl", label: "NiXL" }, + ], + // No IB-device knob: mooncake auto-detects the HCA. Pass + // --disaggregation-ib-device only if discovery picks the wrong NIC + // (see Configuration Tips). + // Router fronting the prefill + decode roles; substitute /. + router: { + port: 8000, + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} \\ + --decode http://:{{DECODE_PORT}} \\ + --host 0.0.0.0 --port {{ROUTER_PORT}} \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`, + }, + }, + + // ----- Card 6: "Hierarchical KV Cache" ----- + hicache: { + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + ], + }, + }, + + cells: [ + // ==================================================================== + // H200 + FP8 (Hopper) — TP8. + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Large chunked-prefill is the main high-throughput tuning lever; + // max-running should track the available KV capacity. + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B200 + FP8 (Blackwell) — TP8. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Large chunked-prefill is the main high-throughput tuning lever; + // max-running should track the available KV capacity. + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 + FP8 (Grace-Blackwell, 4-GPU single node) — TP4. + // The flags follow the B200 recipe with TP4/DP4 for a 4-GPU node. + // Stage the weights on node-local NVMe first — shared cluster-storage reads are slow. + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--dp 4", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + // Same prefill lever as H200/B200; max-running tracks the TP4 KV capacity. + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 + FP8 (Blackwell Ultra, 8-GPU single node) — TP8. + // The recipe follows the B200 FP8 path. + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.8", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dp 8", + "--enable-dp-attention", + "--moe-a2a-backend deepep", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // B300 + BF16 (Blackwell Ultra, 8-GPU single node) — TP8. + // The unquantized GLM-5.3 (~700B, ~1.51 TB) only fits single-node on 8xB300 + // (~2.1 TB HBM); smaller GPUs need multi-node (e.g. 2x 8xH200). Balanced/HT run plain TP8 + // without DP-Attention or DeepEP. + // ==================================================================== + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.9", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.9", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--mem-fraction-static 0.9", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // BF16 multi-node (inferred) — the 1.51 TB checkpoint spread over 2 nodes. + // 2x 8xH200 / 2x 8xB200 at TP16, 2x 4xGB300 at TP8. The engine injects + // --nnodes / --node-rank / --dist-init-addr from the Multi-Nodes selector. + // Recipes reuse the single-node B300 flags and remain unverified. + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 1", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 2", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 32768", + "--max-running-requests 80", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--mem-fraction-static 0.85", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + + // ==================================================================== + // NVFP4 — RadixArk/GLM-5.3-NVFP4 (Model Optimizer, experts-only W4A4). + // Same runtime contract as nvidia/GLM-5.2-NVFP4 (experts-only NVFP4, KV FP8, + // no per-tensor k/v scale tensors), so the GLM-5.2 NVFP4 recipes carry over: + // TP8 on B200/B300, TP4 on GB300; low-latency MTP 5-1-6, balanced MTP 2-1-3. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--dp 8", + "--enable-dp-attention", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.92", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--dp 8", + "--enable-dp-attention", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.92", + "--max-running-requests 512", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--max-running-requests 16", + "--cuda-graph-max-bs 16", + "--max-prefill-tokens 8192", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--dp 8", + "--enable-dp-attention", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.92", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 5", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 6", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.85", + "--max-running-requests 16", + "--cuda-graph-max-bs 16", + "--max-prefill-tokens 8192", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" }, + verified: true, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--dp 4", + "--enable-dp-attention", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 2", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 3", + "--chunked-prefill-size 8192", + "--mem-fraction-static 0.92", + "--max-running-requests 256", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // AMD MI300X / MI325X / MI355X (ROCm) — TP8, DSA tilelang backend. + // No MTP: disabled in the Speculative card for AMD (the gfx950 spec-decode + // draft kernel is not yet validated, and num-steps>3 hits a separate build + // issue). Strategies differ only by batch-shaping levers + // (cuda-graph-max-bs / max-running-requests / chunked-prefill): + // low-latency — large chunked-prefill, default bs. + // balanced — chunked-prefill 32768 + bs128, max-running 80. + // high-throughput — bs256, max-running 256. + // BF16 (~1.51 TB) only fits single-node on MI325X (2 TB) / MI355X (2.3 TB); + // MI300X (1.5 TB) needs multi-node, so its BF16 cells are omitted. + // ==================================================================== + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 131072", + "--mem-fraction-static 0.80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--chunked-prefill-size 32768", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 128", + "--max-running-requests 80", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" }, + verified: false, + env: [], + flags: [ + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dsa-prefill-backend tilelang", + "--dsa-decode-backend tilelang", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--max-running-requests 256", + "--watchdog-timeout 1200", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +};