docs(cookbook): verify GLM-5.2 single-node B300 (FP8 + BF16) (#28460)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
Xinyuan Tong
co-authored by
Zijie Xia
Zijie Xia
parent
c17190c059
commit
72ccfec594
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.2
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, and GB300."
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, and GB300."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -94,10 +94,10 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
- **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required.
|
||||
- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.
|
||||
- **Context Parallelism (CP) for long prefill.** DSA prefill CP splits the long-prefill attention across `--attn-cp-size` ranks. On **Hopper (H200)** this gives a large prefill-latency win at long context — e.g. round-robin CP (`--tp 8 --attn-cp-size 8 --enable-dsa-prefill-context-parallel --dsa-prefill-cp-mode round-robin-split`) cut 64K-token prefill TTFT roughly **2.5–2.8×** vs. plain TP8 in our testing. Trade-offs: CP partitions the KV pool (lower max context at the same `--mem-fraction-static`) and adds some decode-side overhead, so it pays off only for long sequences. **CP is currently verified on Hopper only** — the Blackwell (sm100) DSA-CP FP8 rope kernel is not yet adapted, so leave CP off on B200/GB300.
|
||||
- **Context Parallelism (CP) for long prefill.** DSA prefill CP splits the long-prefill attention across `--attn-cp-size` ranks. On **Hopper (H200)** this gives a large prefill-latency win at long context — e.g. round-robin CP (`--tp 8 --attn-cp-size 8 --enable-dsa-prefill-context-parallel --dsa-prefill-cp-mode round-robin-split`) cut 64K-token prefill TTFT roughly **2.5–2.8×** vs. plain TP8 in our testing. Trade-offs: CP partitions the KV pool (lower max context at the same `--mem-fraction-static`) and adds some decode-side overhead, so it pays off only for long sequences. **CP is currently verified on Hopper only** — the Blackwell (sm100) DSA-CP FP8 rope kernel is not yet adapted, so leave CP off on B200/B300/GB300.
|
||||
- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
|
||||
- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
|
||||
- **BF16 weights need more GPUs (unverified).** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). The BF16 recipes in the panel are **proposed/inferred, not yet benchmarked** (`verified: false`) — FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8.
|
||||
- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.
|
||||
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
@@ -100,14 +100,67 @@ export const benchmarks = [
|
||||
ttft_ms: 250727, tpot_ms: 68.55, tokens_per_sec_per_gpu: 641 },
|
||||
],
|
||||
},
|
||||
// ---- B300 + FP8 ---- (inferred from B200; benchmarks pending → render "pending")
|
||||
{ match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" } },
|
||||
// ---- B300 + BF16 ---- (unquantized zai-org/GLM-5.2; benchmarks pending → render "pending")
|
||||
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } },
|
||||
// ---- B300 + FP8 ---- (8-GPU single node, TP8; measured on v0.5.13.post1, flush-cache every run.
|
||||
// B300 (sm103) trails B200 (sm100) per-GPU here — the deep_gemm/DSA kernels are tuned for sm100 and
|
||||
// fall to a slower path on sm103; the gap should close as sm103 gets first-class kernels.)
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 503, tpot_ms: 3.24, tokens_per_sec_per_gpu: 34 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 4731, tpot_ms: 9.56, tokens_per_sec_per_gpu: 140 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 6465, tpot_ms: 23.36, tokens_per_sec_per_gpu: 245 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 67814, tpot_ms: 26.19, tokens_per_sec_per_gpu: 265 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 206246, tpot_ms: 56.11, tokens_per_sec_per_gpu: 388 },
|
||||
],
|
||||
},
|
||||
// ---- B300 + BF16 ---- (unquantized zai-org/GLM-5.2, TP8; measured on v0.5.13.post1, flush-cache every run.
|
||||
// balanced/HT run plain TP8 (no DP-Attention/DeepEP), so they trail the FP8 dp-attention recipe at high concurrency.)
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 470, tpot_ms: 2.93, tokens_per_sec_per_gpu: 37 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 3474, tpot_ms: 10.33, tokens_per_sec_per_gpu: 146 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 14123, tpot_ms: 35.47, tokens_per_sec_per_gpu: 157 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 116633, tpot_ms: 40.65, tokens_per_sec_per_gpu: 167 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.13.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 525108, tpot_ms: 82.52, tokens_per_sec_per_gpu: 168 },
|
||||
],
|
||||
},
|
||||
// ---- BF16 multi-node (inferred) ---- benchmarks pending
|
||||
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "multi-2" } },
|
||||
{ match: { hw: "h200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" } },
|
||||
|
||||
@@ -354,13 +354,13 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B300 + FP8 (Blackwell Ultra, 8-GPU single node) — TP8.
|
||||
// Inferred from the verified B200 (sm100) FP8 recipe; B300 is the same Blackwell
|
||||
// family (sm103). Benchmarks pending → verified:false.
|
||||
// B300 + FP8 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1).
|
||||
// Recipe mirrors the verified B200 (sm100) FP8 path. B300 (sm103) currently trails B200 per-GPU
|
||||
// because deep_gemm/DSA are tuned for sm100; expected to improve as sm103 gets first-class kernels.
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
@@ -377,7 +377,7 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
@@ -399,7 +399,7 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
@@ -416,14 +416,14 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
|
||||
// ====================================================================
|
||||
// B300 + BF16 (Blackwell Ultra, 8-GPU single node) — TP8.
|
||||
// B300 + BF16 (Blackwell Ultra, 8-GPU single node) — TP8. Verified on 8xB300 (v0.5.13.post1).
|
||||
// The unquantized GLM-5.2 (~700B, ~1.51 TB) only fits single-node on 8xB300
|
||||
// (~2.1 TB HBM); smaller GPUs need multi-node (e.g. 2x 8xH200). Recipes are
|
||||
// proposed, single-node TP8; benchmarks pending → verified:false.
|
||||
// (~2.1 TB HBM); smaller GPUs need multi-node (e.g. 2x 8xH200). balanced/HT run plain TP8
|
||||
// (no DP-Attention/DeepEP), so they trail the FP8 recipe at high concurrency.
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
@@ -440,7 +440,7 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
@@ -459,7 +459,7 @@ sgl-eval run aime25 \\
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
|
||||
verified: false,
|
||||
verified: true,
|
||||
env: [],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
|
||||
Reference in New Issue
Block a user