minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

GLM-5.3-Flash cookbook: default Blackwell recipes to FP8 KV + TRT-LLM DSA (#36519)

Browse Source
This commit is contained in:
Xinyuan Tong
2026-08-27 00:51:05 +08:00
committed by GitHub
parent f8cc1f9525
commit e27a7fac77
4 changed files with 74 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx
+3 -3
View File
@@ -25,7 +25,7 @@ Choose your hardware, then choose the operating point that matches your workload
- **Low Latency** starts with adaptive MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
- **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to **Not Verified** without hiding the option.
@@ -65,7 +65,7 @@ GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a h
</tr>
<tr>
<td style={{padding: "9px 12px"}}>Precision</td>
<td style={{padding: "9px 12px"}}>FP8 weights with BF16 KV cache by default</td>
<td style={{padding: "9px 12px"}}>FP8 weights; FP8 KV cache by default on Blackwell, BF16 KV cache on H100 and H200</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Context</td>
@@ -102,7 +102,7 @@ Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not o
### Keep the KV and DSA backends paired
The default recipe pairs a BF16 KV cache with TileLang DSA. On Blackwell GPUs, the deployment panel also offers FP8 KV with TRT-LLM DSA as a memory-saving option. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination. FP8 KV with TRT-LLM DSA is disabled on H100 and H200.
On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.3–5.5% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination.
### Extend the cache hierarchy
docs/src/snippets/_deployment.jsx
+1
View File
@@ -2506,6 +2506,7 @@ export const Deployment = ({ config, benchmarks }) => {
sel.variant,
sel.quant && sel.quant.toUpperCase(),
sel.strategy,
sel.kvDsaPair,
sel.nodes,
]
.filter((part) => part !== undefined && part !== null && part !== "")
docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx
+10 -9
View File
@@ -12,14 +12,14 @@ export const benchmarks = [
max_concurrency: 16,
num_prompts: 80,
},
ttft_ms: 589.2,
ttft_ms: 599.91,
tpot_ms: 6.48,
tokens_per_sec_per_gpu: 2277.46,
tokens_per_sec_per_gpu: 2266.08,
},
],
accuracy: { gsm8k_pct: 97.50 },
notes:
"Measured on 4x GB300 (TP4/EP4) with the final weights (zai-org/GLM-5.3-Flash, c5b82b63e37b) at the rc2 cut (f13cb6f6a7), adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3 (accept length confirmed 3.00 in the bench summary and server log): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,821.97 aggregate output tok/s after two discarded warmups. Simulated accept length makes this a throughput-mechanism number. Accuracy is from the shared non-simulated full GSM8K gate: 97.50% with a 100% stop rate over all 1,319 problems.",
"Measured on 4x GB300 (TP4/EP4) with the final weights (zai-org/GLM-5.3-Flash, c5b82b63e37b) at the rc2 cut (f13cb6f6a7), adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3 (accept length confirmed 3.00 in the bench summary and server log): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,812.86 aggregate output tok/s after two discarded warmups. Simulated accept length makes this a throughput-mechanism number. Accuracy is from the shared non-simulated full GSM8K gate: 97.50% with a 100% stop rate over all 1,319 problems.",
},
{
match: { hw: "gb300", strategy: "low-latency", kvDsaPair: "fp8-trtllm" },
@@ -34,13 +34,13 @@ export const benchmarks = [
max_concurrency: 16,
num_prompts: 80,
},
ttft_ms: 583.36,
tpot_ms: 6.21,
tokens_per_sec_per_gpu: 2357.1,
ttft_ms: 604.2,
tpot_ms: 6.19,
tokens_per_sec_per_gpu: 2338.53,
},
],
notes:
"The Low Latency recipe with FP8 KV + TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) at rc2 (f13cb6f6a7), adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3 (accept 3.00): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,885.68 aggregate output tok/s — 3.5% above the BF16 + TileLang Low Latency row, with mean TPOT 6.21 ms vs 6.48 ms. Draft and target full-graph capture succeeded for this combination.",
"The Low Latency recipe with FP8 KV + TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) at rc2 (f13cb6f6a7), adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3 (accept 3.00): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,870.82 aggregate output tok/s — 3.2% above the BF16 + TileLang Low Latency row, with mean TPOT 6.19 ms vs 6.48 ms. Draft and target full-graph capture succeeded for this combination.",
},
{
match: { hw: "gb300", strategy: "high-throughput" },
@@ -86,7 +86,7 @@ export const benchmarks = [
],
accuracy: { gsm8k_pct: 97.50 },
notes:
"Measured on 4x GB300 (TP4/EP4) with the final weights (zai-org/GLM-5.3-Flash, c5b82b63e37b) at the rc2 cut (f13cb6f6a7), speculative decoding off, after two discarded warmups per row: 1,128.32 / 2,418.65 / 3,884.95 aggregate output tok/s at concurrency 16 / 64 / 256 (80 / 320 / 1,280 random requests at 1,024 input / 256 output tokens; the high-concurrency rows with --max-running-requests 256 and decode graph batch 256). Throughput at 256 is still scaling but sublinear (prefill queueing). Accuracy is the full GSM8K gate on the same server: 97.50% with a 100% stop rate over all 1,319 problems.",
"Measured on 4x GB300 (TP4/EP4) with the final weights (zai-org/GLM-5.3-Flash, c5b82b63e37b) at the rc2 cut (f13cb6f6a7), speculative decoding off, after two discarded warmups per row: 1,128.32 / 2,418.65 / 3,884.95 aggregate output tok/s at concurrency 16 / 64 / 256 (80 / 320 / 1,280 random requests at 1,024 input / 256 output tokens). The measured server added --max-running-requests 256, --cuda-graph-backend-decode full, --cuda-graph-max-bs-decode 256, --chunked-prefill-size 8192, --max-prefill-tokens 8192, and --disable-shared-experts-fusion to the cell command — the concurrency 64/256 rows are not reproducible at the cell's automatic sizing. Throughput at 256 is still scaling but sublinear (prefill queueing). Accuracy is the full GSM8K gate on the same server: 97.50% with a 100% stop rate over all 1,319 problems. With HiCache L1+L2 (32 GB host tier) the same protocol measured 1,125.45 / 2,403.57 / 3,839.57 tok/s — a 0.3-1.2% overhead; the random dataset has no prefix reuse, so L2 benefit was not exercised.",
},
{
match: { hw: "gb300", strategy: "high-throughput", kvDsaPair: "fp8-trtllm" },
@@ -130,8 +130,9 @@ export const benchmarks = [
tokens_per_sec_per_gpu: 4965.65,
},
],
accuracy: { gsm8k_pct: 97.35 },
notes:
"FP8 KV cache with TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) at rc2 (f13cb6f6a7), same protocol as the BF16 rows: 1,189.96 / 2,476.87 / 3,972.52 aggregate output tok/s at concurrency 16 / 64 / 256 — 2.3–5.5% above BF16 + TileLang across the curve, and the FP8 pool holds 13.5M tokens per rank vs 7.5M at BF16 (1.8x capacity at identical pool bytes). Sanity requests answered correctly and stopped cleanly; accuracy was not re-run for this variant (the 97.50% GSM8K gate used BF16 KV).",
"FP8 KV cache with TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) at rc2 (f13cb6f6a7), same protocol as the BF16 rows: 1,189.96 / 2,476.87 / 3,972.52 aggregate output tok/s at concurrency 16 / 64 / 256 — 2.3–5.5% above BF16 + TileLang across the curve, and the FP8 pool holds 13.5M tokens per rank vs 7.5M at BF16 (1.8x capacity at identical pool bytes). Accuracy is the full GSM8K gate on this variant: 97.35% vs 97.50% on BF16 KV, a 0.15-point gap inside sampling noise, with a 99.92% stop rate (one truncated run of 1,319). With HiCache L1+L2 (32 GB host tier) the same protocol measured 1,187.13 / 2,464.89 / 3,941.37 tok/s — a 0.2-0.8% overhead; the random dataset has no prefix reuse, so L2 benefit was not exercised.",
},
{ match: { hw: "h100", strategy: "low-latency" } },
{ match: { hw: "h100", strategy: "high-throughput" } },
docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
+60 -41
View File
@@ -15,8 +15,9 @@ export const config = {
],
isRecommendedSelection(s) {
const pairing = ["h100", "h200"].includes(s.hw) ? "bf16-tilelang" : "fp8-trtllm";
return (
s.kvDsaPair === "bf16-tilelang" &&
s.kvDsaPair === pairing &&
s.mmTransport === "auto" &&
s.hicache === "off"
);
@@ -26,18 +27,8 @@ export const config = {
{
id: "kvDsaPair",
title: "KV Cache + DSA Backend",
default: "bf16-tilelang",
default: "fp8-trtllm",
options: [
{
id: "bf16-tilelang",
label: "BF16 + TileLang",
stripPrefixes: ["--kv-cache-dtype", "--dsa-prefill-backend", "--dsa-decode-backend"],
flags: [
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
],
},
{
id: "fp8-trtllm",
label: "FP8 + TRT-LLM",
@@ -49,7 +40,17 @@ export const config = {
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
],
hints: ["Reduces KV-cache memory. Validate accuracy and memory headroom for your workload."],
hints: ["Measured on GB300: faster than BF16 + TileLang with about 1.8x the KV token capacity."],
},
{
id: "bf16-tilelang",
label: "BF16 + TileLang",
stripPrefixes: ["--kv-cache-dtype", "--dsa-prefill-backend", "--dsa-decode-backend"],
flags: [
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
],
},
],
},
@@ -78,6 +79,8 @@ export const config = {
label: "L1 + L2",
subtitle: "Host memory",
flags: ["--enable-hierarchical-cache", "--hicache-size 32"],
disabled: (s) => s.strategy === "low-latency",
disableReason: "HiCache with MTP speculative decoding crashes at startup in the current build (DSA draft pool lacks full_kv_pool); use it with High Throughput only.",
hints: ["32 GB host tier; the default ratio can demand more host RAM than the node has free."],
},
{
@@ -86,6 +89,8 @@ export const config = {
subtitle: "Mooncake",
flags: ["--enable-hierarchical-cache", "--hicache-size 32", "--hicache-storage-backend mooncake"],
env: ["SGLANG_HICACHE_MOONCAKE_CONFIG_PATH={{MOONCAKE_CONFIG}}"],
disabled: (s) => s.strategy === "low-latency",
disableReason: "HiCache with MTP speculative decoding crashes at startup in the current build (DSA draft pool lacks full_kv_pool); use it with High Throughput only.",
hints: ["Start Mooncake and place the configuration file on every serving node."],
},
],
@@ -240,8 +245,9 @@ sgl-eval run gsm8k \\
nnodes: 1,
verified: true,
verificationStatus: (s) =>
config.isRecommendedSelection(s) ||
(s.kvDsaPair === "fp8-trtllm" && s.mmTransport === "auto" && s.hicache === "off")
["bf16-tilelang", "fp8-trtllm"].includes(s.kvDsaPair) &&
s.mmTransport === "auto" &&
s.hicache === "off"
? "verified"
: "unverified",
env: [],
@@ -249,10 +255,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -269,8 +276,9 @@ sgl-eval run gsm8k \\
nnodes: 1,
verified: true,
verificationStatus: (s) =>
config.isRecommendedSelection(s) ||
(s.kvDsaPair === "fp8-trtllm" && s.mmTransport === "auto" && s.hicache === "off")
["bf16-tilelang", "fp8-trtllm"].includes(s.kvDsaPair) &&
s.mmTransport === "auto" &&
s.hicache === "off"
? "verified"
: "unverified",
env: [],
@@ -278,10 +286,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -303,6 +312,7 @@ sgl-eval run gsm8k \\
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -328,6 +338,7 @@ sgl-eval run gsm8k \\
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -347,6 +358,7 @@ sgl-eval run gsm8k \\
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -371,6 +383,7 @@ sgl-eval run gsm8k \\
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -386,10 +399,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -410,10 +424,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -429,10 +444,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -453,10 +469,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
"--ep-size 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -472,10 +489,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
@@ -496,10 +514,11 @@ sgl-eval run gsm8k \\
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
"--ep-size 4",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--kv-cache-dtype bfloat16",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
"--moe-runner-backend deep_gemm",
"--disable-shared-experts-fusion",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",