[Docs] Add GLM-5.3 cookbook (#36827)
Co-authored-by: JustinTong0323 <xinyuantong.cs@gmail.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai> Co-authored-by: Mohammad Angkad <176301910+mmangkad@users.noreply.github.com>
This commit is contained in:
Baizhou Zhang
co-authored by
JustinTong0323
Zijie Xia
Mohammad Angkad
JustinTong0323
Zijie Xia
Mohammad Angkad
parent
b644771e07
commit
395c2258c3
@@ -0,0 +1,259 @@
|
|||||||
|
---
|
||||||
|
title: GLM-5.3
|
||||||
|
description: "Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||||
|
tag: NEW
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
<a id="install" />
|
||||||
|
|
||||||
|
<Accordion title="Install SGLang">
|
||||||
|
|
||||||
|
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab title="Python (pip / uv)">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
pip install --upgrade pip
|
||||||
|
pip install uv
|
||||||
|
uv pip install --prerelease=allow sglang
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the **Python** output of the command panel below in that environment.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab title="Docker">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
docker pull lmsysorg/sglang:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||||
|
|
||||||
|
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||||
|
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||||
|
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||||
|
|
||||||
|
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||||
|
import { config } from "/src/snippets/configs/zai-org/glm-5.3.jsx";
|
||||||
|
import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx";
|
||||||
|
|
||||||
|
<Deployment config={config} benchmarks={benchmarks} />
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.3.
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
## Playground
|
||||||
|
|
||||||
|
The Playground is where you experiment with **SGLang features beyond the deployment matrix**. It lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||||
|
|
||||||
|
import { Playground } from "/src/snippets/_playground.jsx";
|
||||||
|
|
||||||
|
<Playground config={config} />
|
||||||
|
|
||||||
|
## 1. Model Introduction
|
||||||
|
|
||||||
|
**GLM-5.3** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It keeps the same base architecture while updating post-training for complex coding and long-horizon tasks. It ships in two precisions — **FP8** (`zai-org/GLM-5.3`) and full **BF16** (`zai-org/GLM-5.3-BF16`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, RadixArk publishes an **experimental NVFP4** build (`RadixArk/GLM-5.3-NVFP4`, Model Optimizer) that quantizes only the routed experts' linear weights and activations to 4-bit (attention, shared experts, dense layers, MTP, embeddings, and the LM head stay unquantized), cutting the weight footprint to ~0.45 TB so a 4-GPU GB300 node can serve it with TP4. **The NVFP4 cells are experimental** — benchmark data (accuracy / throughput) is still pending; treat them as provisional until the benchmark cards fill in.
|
||||||
|
|
||||||
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||||
|
<thead>
|
||||||
|
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Context</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.3">GLM-5.3</a></strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
|
||||||
|
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.3-BF16">GLM-5.3-BF16</a></strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
|
||||||
|
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/RadixArk/GLM-5.3-NVFP4">GLM-5.3-NVFP4</a></strong></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
|
||||||
|
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).
|
||||||
|
|
||||||
|
**Resources:** [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) · [GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16) · [GLM-5.3-NVFP4](https://huggingface.co/RadixArk/GLM-5.3-NVFP4).
|
||||||
|
|
||||||
|
## 2. Configuration Tips
|
||||||
|
|
||||||
|
- **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.
|
||||||
|
- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work.
|
||||||
|
- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
|
||||||
|
- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
|
||||||
|
- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8.
|
||||||
|
- **PD Disaggregation (prefill/decode).** GLM-5.3 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.
|
||||||
|
- **Chunked-prefill size is regime-dependent.** For long-input balanced workloads, start with `--chunked-prefill-size 32768` and tune it together with `--max-running-requests` for your input length and KV capacity. Keep the default chunked-prefill size for the high-throughput recipe unless profiling shows that prefill is the bottleneck.
|
||||||
|
|
||||||
|
- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.3`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.3-BF16`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there. Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). GLM-5.3 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).
|
||||||
|
- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel because the gfx950 spec-decode draft kernel is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.
|
||||||
|
|
||||||
|
## 3. Advanced Usage
|
||||||
|
|
||||||
|
### 3.1 Reasoning
|
||||||
|
|
||||||
|
GLM-5.3 is a reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response.
|
||||||
|
|
||||||
|
**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to select `low`, `high`, or `max`. If you omit it or pass another value, the template uses `max`.
|
||||||
|
|
||||||
|
| `reasoning_effort` | Injected system line | Effect |
|
||||||
|
|---|---|---|
|
||||||
|
| *(not passed / unset)*, `"max"` | `Reasoning Effort: Max` | default — highest reasoning effort |
|
||||||
|
| `"high"` | `Reasoning Effort: High` | high reasoning effort |
|
||||||
|
| `"low"` | `Reasoning Effort: Low` | low reasoning effort |
|
||||||
|
| any other value | `Reasoning Effort: Max` | falls back to the default |
|
||||||
|
|
||||||
|
<Accordion title="Reasoning Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
resp = client.chat.completions.create(
|
||||||
|
model="zai-org/GLM-5.3",
|
||||||
|
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||||
|
extra_body={"chat_template_kwargs": {"clear_thinking": True, "reasoning_effort": "high"}},
|
||||||
|
)
|
||||||
|
msg = resp.choices[0].message
|
||||||
|
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||||
|
print("Answer:", msg.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Example Output">
|
||||||
|
|
||||||
|
```text Output
|
||||||
|
Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240.
|
||||||
|
2. **Convert the percentage to a decimal:** 15% = 0.15
|
||||||
|
3. **Multiply by the total:** 0.15 * 240 = 36
|
||||||
|
(Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
|
||||||
|
|
||||||
|
Answer: 15% of 240 is **36**.
|
||||||
|
|
||||||
|
Here is how you can calculate it:
|
||||||
|
0.15 × 240 = 36
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
### 3.2 Tool Calling
|
||||||
|
|
||||||
|
Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.3 emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields.
|
||||||
|
|
||||||
|
<Accordion title="Tool Calling Example (Python)">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
tools = [{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"description": "Get the current weather for a city",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {"city": {"type": "string"}},
|
||||||
|
"required": ["city"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}]
|
||||||
|
resp = client.chat.completions.create(
|
||||||
|
model="zai-org/GLM-5.3",
|
||||||
|
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||||
|
tools=tools,
|
||||||
|
)
|
||||||
|
msg = resp.choices[0].message
|
||||||
|
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||||
|
print("Tool calls:", msg.tool_calls)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
<Accordion title="Example Output">
|
||||||
|
|
||||||
|
```text Output
|
||||||
|
Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
|
||||||
|
|
||||||
|
Tool calls: [
|
||||||
|
{
|
||||||
|
"id": "call_13fcd52146934b7781d06d4a",
|
||||||
|
"type": "function",
|
||||||
|
"function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
### 3.3 HiCache (Hierarchical KV Caching)
|
||||||
|
|
||||||
|
For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.3's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.
|
||||||
|
|
||||||
|
### 3.4 Claude Code Integration
|
||||||
|
|
||||||
|
GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then:
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
|
||||||
|
export ANTHROPIC_AUTH_TOKEN="dummy"
|
||||||
|
export API_TIMEOUT_MS="3000000"
|
||||||
|
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
|
||||||
|
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
|
||||||
|
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
|
||||||
|
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.3[1m]"
|
||||||
|
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.3[1m]"
|
||||||
|
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.3[1m]"
|
||||||
|
claude
|
||||||
|
```
|
||||||
|
|
||||||
|
Two of these matter specifically for GLM-5.3:
|
||||||
|
|
||||||
|
- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.3's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
|
||||||
|
- **`glm-5.3[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.3's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.
|
||||||
|
|
||||||
|
For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).
|
||||||
|
|
||||||
|
### 3.5 Context Parallelism
|
||||||
|
|
||||||
|
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments:
|
||||||
|
```bash
|
||||||
|
--attn-cp-size 8 \
|
||||||
|
--enable-prefill-cp \
|
||||||
|
--cp-strategy interleave \
|
||||||
|
```
|
||||||
|
which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.
|
||||||
|
|
||||||
|
When deploying with PD Disaggregation, a **prefill worker using the Mooncake transfer backend** can enable the [LayerSplit](https://z.ai/blog/scaling-pain) technique with
|
||||||
|
```bash
|
||||||
|
--enable-dsa-cache-layer-split \
|
||||||
|
--enable-prefill-cp \
|
||||||
|
--attn-cp-size 8 \
|
||||||
|
--cp-strategy interleave \
|
||||||
|
```
|
||||||
|
With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.
|
||||||
@@ -1226,6 +1226,7 @@
|
|||||||
{
|
{
|
||||||
"group": "GLM",
|
"group": "GLM",
|
||||||
"pages": [
|
"pages": [
|
||||||
|
"cookbook/autoregressive/GLM/GLM-5.3",
|
||||||
"cookbook/autoregressive/GLM/GLM-5.3-Flash",
|
"cookbook/autoregressive/GLM/GLM-5.3-Flash",
|
||||||
"cookbook/autoregressive/GLM/GLM-5.2",
|
"cookbook/autoregressive/GLM/GLM-5.2",
|
||||||
"cookbook/autoregressive/GLM/GLM-5.1",
|
"cookbook/autoregressive/GLM/GLM-5.1",
|
||||||
|
|||||||
@@ -0,0 +1,221 @@
|
|||||||
|
// GLM-5.3 benchmark placeholders, keyed by the same `match` tuple as glm-5.3.jsx cells.
|
||||||
|
// Bare match stubs render as pending until speed and accuracy measurements are available.
|
||||||
|
export const benchmarks = [
|
||||||
|
// NVIDIA FP8, single node.
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.42 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 780, tpot_ms: 3.71, tokens_per_sec_per_gpu: 252 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 5499, tpot_ms: 14.20, tokens_per_sec_per_gpu: 920 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.496 at concurrency 1, 3.476 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.12, aime25_pct: 91.88 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 7790, tpot_ms: 21.90, tokens_per_sec_per_gpu: 2440 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 57607, tpot_ms: 33.39, tokens_per_sec_per_gpu: 2558 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.12 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 400, tpot_ms: 3.11, tokens_per_sec_per_gpu: 321 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 3044, tpot_ms: 7.84, tokens_per_sec_per_gpu: 1662 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.507 at concurrency 1, 3.503 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime25_pct: 90.83 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 5908, tpot_ms: 16.53, tokens_per_sec_per_gpu: 3220 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 18916, tpot_ms: 31.88, tokens_per_sec_per_gpu: 5025 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.73 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 372, tpot_ms: 3.68, tokens_per_sec_per_gpu: 556 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 3305, tpot_ms: 10.02, tokens_per_sec_per_gpu: 2718 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.489 at concurrency 1, 3.509 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.19, aime25_pct: 90.62 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 7775, tpot_ms: 22.77, tokens_per_sec_per_gpu: 4743 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 25775, tpot_ms: 45.87, tokens_per_sec_per_gpu: 7501 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.12 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 342, tpot_ms: 3.10, tokens_per_sec_per_gpu: 328 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 2851, tpot_ms: 7.77, tokens_per_sec_per_gpu: 1705 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.485 at concurrency 1, 3.487 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime25_pct: 92.08 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 5586, tpot_ms: 16.37, tokens_per_sec_per_gpu: 3296 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 17848, tpot_ms: 32.48, tokens_per_sec_per_gpu: 5259 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
|
||||||
|
// NVIDIA NVFP4, single node (RadixArk/GLM-5.3-NVFP4).
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 26fd7fdaa273",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 281, tpot_ms: 1.48, tokens_per_sec_per_gpu: 640 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 1988, tpot_ms: 5.11, tokens_per_sec_per_gpu: 2306 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 26fd7fdaa273",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 5227, tpot_ms: 12.40, tokens_per_sec_per_gpu: 3748 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 17513, tpot_ms: 29.97, tokens_per_sec_per_gpu: 5416 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 26fd7fdaa273",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 307, tpot_ms: 1.71, tokens_per_sec_per_gpu: 1113 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 2132, tpot_ms: 6.18, tokens_per_sec_per_gpu: 3979 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 26fd7fdaa273",
|
||||||
|
accuracy: { gsm8k_pct: 97.42, aime26_pct: 94.17 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 5828, tpot_ms: 16.69, tokens_per_sec_per_gpu: 6108 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 22330, tpot_ms: 39.63, tokens_per_sec_per_gpu: 8537 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
// NVIDIA BF16.
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.12 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||||
|
ttft_ms: 344, tpot_ms: 2.96, tokens_per_sec_per_gpu: 341 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||||
|
ttft_ms: 2772, tpot_ms: 9.13, tokens_per_sec_per_gpu: 1521 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=3.5 with the EAGLE 5/1/6 draft (measured accept length 3.531 at concurrency 1, 3.506 at concurrency 16). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.27, aime25_pct: 93.75 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||||
|
ttft_ms: 9512, tpot_ms: 22.30, tokens_per_sec_per_gpu: 2279 },
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||||
|
ttft_ms: 75250, tpot_ms: 25.84, tokens_per_sec_per_gpu: 2333 },
|
||||||
|
],
|
||||||
|
notes:
|
||||||
|
"Speed measured under SGLANG_SIMULATE_ACC_LEN=2 with the EAGLE 1/1/2 draft (measured accept length 2.000 exactly at both concurrencies, saturating the 2-token draft). The pinned accept length makes these throughput-mechanism numbers only, never correctness evidence; the accuracy runs carry no such env.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||||
|
sglang_version: "main @ 20a491d1d311",
|
||||||
|
accuracy: { gsm8k_pct: 97.12 },
|
||||||
|
speed: [
|
||||||
|
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||||
|
ttft_ms: 349227, tpot_ms: 65.45, tokens_per_sec_per_gpu: 2216 },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } },
|
||||||
|
{ match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "multi-2" } },
|
||||||
|
|
||||||
|
// AMD ROCm, single node.
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||||
|
{ match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } },
|
||||||
|
];
|
||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user