minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] GLM-5.3-Flash cookbook: add NVFP4 FP8+TRT-LLM benchmark rows (follow-up to #37109) (#37412)

Browse Source
This commit is contained in:
Xinyuan Tong
2026-09-01 12:56:22 -07:00
committed by GitHub
parent 0b1ce3d140
commit 442c7c1e29
2 changed files with 95 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx
+91 -3
View File
@@ -181,6 +181,18 @@ export const benchmarks = [
sglang_version: "033446bb05",
latencyPercentile: "Mean",
speed: [
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 1,
num_prompts: 8,
},
ttft_ms: 148.18,
tpot_ms: 3.84,
tokens_per_sec_per_gpu: 256.69,
},
{
workload: {
dataset: "random",
@@ -196,7 +208,7 @@ export const benchmarks = [
],
accuracy: { gsm8k_pct: 97.14, aime2026_pct: 92.45 },
notes:
"RadixArk/GLM-5.3-Flash-NVFP4 — NVFP4 W4A4 post-training quantization of zai-org/GLM-5.3-Flash-BF16 with NVIDIA Model Optimizer 0.46.0 (abs-max scaling, group size 16): routed and shared experts plus the dense MLPs are FP4, while all attention (KDA, DSA indexer, MLA), the router, norms, the vision tower, the MTP layer, embeddings, and the LM head stay BF16. Measured on 4x GB300 with the lmsysorg/sglang:glm-5.3-flash image (PR #36507 head 033446bb05), adaptive MTP 5/1/6, BF16 KV + TileLang DSA, and the flashinfer_cutlass MoE runner. GSM8K 97.14% over the full 1,319-example split x 4 seeds (per-seed range 96.89-97.42%, stop rate 99.85-100%) and AIME 2026 92.45% (30 problems x 16 repeats x 4 seeds = 1,920 generations, per-seed range 91.67-93.54%), both at temperature 1.0 / top_p 0.95. The accuracy runs used the NEXTN spelling of --speculative-algorithm, which resolves to the same runtime path as the published EAGLE command on this tree. The speed row was measured on the unpatched docker tree with SGLANG_SIMULATE_ACC_LEN=3 pinning the accept length (confirmed 2.98): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 818.09 aggregate output tok/s after two discarded warmups. Simulated accept length makes this a throughput-mechanism number; a same-pod A/B against the SwiGLU-fusion-disabled tree landed within ±3%, so the unpatched numbers are the published ones. The companion GSM8K gate on this stack scored 97.27% with a 99.92% stop rate over all 1,319 problems. The gap to the fp8 rows (~45%) is recipe-level, not kernel-isolated: flashinfer_cutlass + BF16 KV + TileLang DSA + TP4 without EP vs the fp8 cells' deep_gemm + FP8 KV + TRT-LLM DSA + EP4.",
"RadixArk/GLM-5.3-Flash-NVFP4 — NVFP4 W4A4 post-training quantization of zai-org/GLM-5.3-Flash-BF16 with NVIDIA Model Optimizer 0.46.0 (abs-max scaling, group size 16): routed and shared experts plus the dense MLPs are FP4, while all attention (KDA, DSA indexer, MLA), the router, norms, the vision tower, the MTP layer, embeddings, and the LM head stay BF16. Measured on 4x GB300 with the lmsysorg/sglang:glm-5.3-flash image (PR #36507 head 033446bb05), adaptive MTP 5/1/6, BF16 KV + TileLang DSA, and the flashinfer_cutlass MoE runner. GSM8K 97.14% over the full 1,319-example split x 4 seeds (per-seed range 96.89-97.42%, stop rate 99.85-100%) and AIME 2026 92.45% (30 problems x 16 repeats x 4 seeds = 1,920 generations, per-seed range 91.67-93.54%), both at temperature 1.0 / top_p 0.95. The accuracy runs used the NEXTN spelling of --speculative-algorithm, which resolves to the same runtime path as the published EAGLE command on this tree. The speed row was measured on the unpatched docker tree with SGLANG_SIMULATE_ACC_LEN=3 pinning the accept length (confirmed 2.98): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 818.09 aggregate output tok/s after two discarded warmups. Simulated accept length makes this a throughput-mechanism number; a same-pod A/B against the SwiGLU-fusion-disabled tree landed within ±3%, so the unpatched numbers are the published ones. The companion GSM8K gate on this stack scored 97.27% with a 99.92% stop rate over all 1,319 problems. The gap to the fp8 rows (~45%) is recipe-level, not kernel-isolated: flashinfer_cutlass + BF16 KV + TileLang DSA + TP4 without EP vs the fp8 cells' deep_gemm + FP8 KV + TRT-LLM DSA + EP4. The concurrency-1 entry uses the same protocol (8 requests, simulated accept 3.00); the fp8-trtllm pairing is ~7% ahead on decode at c1, consistent with the c16 ranking.",
},
{
match: { hw: "gb300", strategy: "high-throughput", quant: "nvfp4", kvDsaPair: "bf16-tilelang" },
@@ -244,8 +256,84 @@ export const benchmarks = [
notes:
"RadixArk/GLM-5.3-Flash-NVFP4 with speculative decoding off — same checkpoint, image, and 4x GB300 measurement stack as the NVFP4 Low Latency row (ModelOpt 0.46.0 NVFP4 W4A4, abs-max, group size 16; MoE and dense MLPs in FP4, attention/router/MTP/embeddings BF16). Accuracy is a checkpoint-level result carried from that arm: GSM8K 97.14% over the full 1,319-example split x 4 seeds (per-seed range 96.89-97.42%, stop rate 99.85-100%) and AIME 2026 92.45% (30 problems x 16 repeats x 4 seeds, per-seed range 91.67-93.54%). Those runs used the NEXTN spelling of --speculative-algorithm on the adaptive-MTP arm, which resolves to the same runtime path as the published EAGLE command. The speed rows were measured on the unpatched docker tree after two discarded warmups per row: 629.43 / 1,054.14 / 2,015.16 aggregate output tok/s at concurrency 16 / 64 / 256 (80 / 320 / 1,280 random requests at 1,024 input / 256 output tokens), with decode on CUDA graphs through bs256 — the published cell command carries no --cuda-graph-max-bs cap. The companion GSM8K gate on this stack scored 97.27% with a 99.92% stop rate over all 1,319 problems. The gap to the fp8 rows (~40-55%) is recipe-level, not kernel-isolated: flashinfer_cutlass + BF16 KV + TileLang DSA + TP4 without EP vs the fp8 cells' deep_gemm + FP8 KV + TRT-LLM DSA + EP4.",
},
{ match: { hw: "gb300", strategy: "low-latency", quant: "nvfp4", kvDsaPair: "fp8-trtllm" } },
{ match: { hw: "gb300", strategy: "high-throughput", quant: "nvfp4", kvDsaPair: "fp8-trtllm" } },
{
match: { hw: "gb300", strategy: "low-latency", quant: "nvfp4", kvDsaPair: "fp8-trtllm" },
sglang_version: "033446bb05",
latencyPercentile: "Mean",
speed: [
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 1,
num_prompts: 8,
},
ttft_ms: 143.16,
tpot_ms: 3.55,
tokens_per_sec_per_gpu: 274.63,
},
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 16,
num_prompts: 80,
},
ttft_ms: 143.81,
tpot_ms: 12.42,
tokens_per_sec_per_gpu: 1424.79,
},
],
notes:
"FP8 KV + TRT-LLM DSA pairing of the NVFP4 recipe on 4x B300 (SXM6), stock unpatched lmsysorg/sglang:glm-5.3-flash image (tree 033446bb05), TP4-only with the flashinfer_cutlass MoE runner, adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3: 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,139.83 aggregate output tok/s after two discarded warmups — 39% above the bf16-tilelang NVFP4 row on the otherwise identical recipe. Simulated accept length makes this a throughput-mechanism number (accept 2.98 confirms the simulation engaged; smoke output is N/A by design under simulated acceptance). EP is not available for this checkpoint on the stock image: --ep-size 4 crashes on the first forward pass (the shared-expert NVFP4 weight arrives 1-D under EP, ValueError in the modelopt_fp4 apply), so the NVFP4 cells are TP-only for now. The concurrency-1 entry uses the same protocol (8 requests, simulated accept 3.00); it stays ~7% ahead of bf16-tilelang on decode, consistent with the c16 ranking.",
},
{
match: { hw: "gb300", strategy: "high-throughput", quant: "nvfp4", kvDsaPair: "fp8-trtllm" },
sglang_version: "033446bb05",
latencyPercentile: "Mean",
speed: [
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 16,
num_prompts: 80,
},
ttft_ms: 107.02,
tpot_ms: 15.98,
tokens_per_sec_per_gpu: 1079.78,
},
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 64,
num_prompts: 320,
},
ttft_ms: 150.56,
tpot_ms: 36.94,
tokens_per_sec_per_gpu: 1998.81,
},
{
workload: {
dataset: "random",
isl: 1024,
osl: 256,
max_concurrency: 256,
num_prompts: 1280,
},
ttft_ms: 279.77,
tpot_ms: 81.04,
tokens_per_sec_per_gpu: 3750.86,
},
],
notes:
"FP8 KV + TRT-LLM DSA pairing of the NVFP4 recipe — same 4x B300 (SXM6), stock unpatched image (tree 033446bb05), TP4-only flashinfer_cutlass stack as the NVFP4 fp8-trtllm Low Latency row, speculative decoding off, after two discarded warmups per row: 863.82 / 1,599.05 / 3,000.69 aggregate output tok/s at concurrency 16 / 64 / 256 (80 / 320 / 1,280 random requests at 1,024 input / 256 output tokens), 37-52% above the bf16-tilelang NVFP4 rows on the otherwise identical recipe. EP is not available for this checkpoint on the stock image: --ep-size 4 crashes on the first forward pass (the shared-expert NVFP4 weight arrives 1-D under EP), so the NVFP4 cells are TP-only for now.",
},
{
match: { hw: "h100", strategy: "low-latency" },
sglang_version: "f040cc72e6",
docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
+4 -8
View File
@@ -451,8 +451,10 @@ sgl-eval run gsm8k \\
],
},
// Same NVFP4 recipe on the remaining Blackwell platforms, at each fp8
// cell's parallelism (gb200 TP4/EP4, b200/b300 TP8/EP8). Not measured on
// this hardware, so every cell here reports unverified.
// cell's TP size (gb200 TP4, b200/b300 TP8). Not measured on this
// hardware, so every cell here reports unverified. All NVFP4 cells are
// TP-only: --ep-size crashes for this checkpoint on the stock image (the
// shared-expert NVFP4 weight arrives 1-D under EP).
{
match: { hw: "gb200", strategy: "low-latency", quant: "nvfp4" },
nnodes: 1,
@@ -462,7 +464,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
@@ -489,7 +490,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
@@ -510,7 +510,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
@@ -537,7 +536,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
@@ -558,7 +556,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
@@ -585,7 +582,6 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--quantization modelopt_fp4",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",