diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx
index 8b220890a..c333f813b 100644
--- a/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx
+++ b/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx
@@ -22,7 +22,7 @@ The deployment panel can render a complete `docker run` command for the selected
Choose your hardware, then choose the operating point that matches your workload:
-- **Low Latency** starts with adaptive MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
+- **Low Latency** starts with MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
- **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
@@ -94,7 +94,7 @@ The deployment recipes use the checkpoint's generation configuration. Override s
### Choose a strategy
-Start with **Low Latency** for chat and agent workloads. Adaptive MTP changes the draft depth as acceptance changes, reducing unnecessary draft work when the server is busy. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value.
+Start with **Low Latency** for chat and agent workloads. It drafts from the checkpoint's MTP head at a fixed depth (5 steps, top-k 1, 6 draft tokens) with natural acceptance. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value.
Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes.
@@ -102,7 +102,7 @@ Strategy labels describe the workload goal, not a hardware restriction. Both str
The **Speculative** card in the Playground changes the algorithm without leaving the selected strategy:
-- **EAGLE / Adaptive MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head.
+- **EAGLE / MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head.
- **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from.
- **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. It needs a build that carries the GLM-5.3-Flash hidden-state capture from [PR #36708](https://github.com/sgl-project/sglang/pull/36708), which is merged into the [PR #36507](https://github.com/sgl-project/sglang/pull/36507) support branch (`xinyuan/glm-5.3-flash-support`) rather than into `main`, so the image pinned above is not enough on its own — pull that branch at its current head, or add #36708's commit on top of an older checkout. The draft repository is also access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point.
@@ -122,7 +122,7 @@ On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300
### Decode context parallelism
-Decode context parallelism (DCP) shards the KV across GPUs during decode to cut long-context latency. The **Context Parallelism** row offers DCP4, validated on 4x GB300 (TP4/EP4) with both KV pairings and adaptive MTP 5/1/6; it requires the current release image, which carries the TileLang LSE fix. Other platforms and attention backends are unvalidated, and draft-extend v2 is unsupported under DCP.
+Decode context parallelism (DCP) shards the KV across GPUs during decode to cut long-context latency. The **Context Parallelism** row offers DCP4, validated on 4x GB300 with both KV pairings on the earlier TP4/EP4 recipe and queued for re-measurement on the current command; it requires the current release image, which carries the TileLang LSE fix. Other platforms and attention backends are unvalidated, and draft-extend v2 is unsupported under DCP.
### Extend the cache hierarchy
@@ -152,7 +152,7 @@ The base recipes accept image and video content through the OpenAI-compatible ch
### 3.4 Encoder disaggregation
-Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4/EP4 process on port 30000. Start the encoder first.
+Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4 process on port 30000. Start the encoder first.
@@ -167,20 +167,20 @@ sglang serve \
+{/* TODO: re-validate encoder disaggregation with this language-server command; the 238,080-visual-token run used the earlier TP4/EP4 deep_gemm command with adaptive MTP. */}
```bash Command
sglang serve \
--model-path zai-org/GLM-5.3-Flash \
--tp-size 4 \
- --ep-size 4 \
--attention-backend dsa \
--dsa-prefill-backend tilelang \
--dsa-decode-backend tilelang \
--linear-attn-backend triton \
--kv-cache-dtype bfloat16 \
--quantization fp8 \
- --moe-runner-backend deep_gemm \
+ --moe-runner-backend flashinfer_trtllm \
--max-running-requests 64 \
--chunked-prefill-size 8192 \
--max-prefill-tokens 8192 \
@@ -189,7 +189,6 @@ sglang serve \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
- --speculative-adaptive \
--language-only \
--encoder-urls http://localhost:30001 \
--mem-fraction-static 0.78 \
@@ -261,7 +260,7 @@ Operational notes:
- Give each role a distinct `--nccl-port` when both share one node.
- Single-node NIXL needs `UCX_NET_DEVICES=lo` and `UCX_TLS=tcp,cuda_copy,cuda_ipc,self,sm` in both server environments.
-- The validated arm used the triton MoE runner; deep_gemm under PD is untested.
+- The validated arm used the triton MoE runner; the flashinfer_trtllm runner from the deployment recipes is untested under PD.
Known limitations:
diff --git a/docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx b/docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx
index 3220ae8a4..0277f35d9 100644
--- a/docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx
+++ b/docs/src/snippets/configs/zai-org/glm-5.3-flash-benchmarks.jsx
@@ -1,4 +1,5 @@
export const benchmarks = [
+ // TODO: speed measured on the earlier TP4/EP4 deep_gemm command with adaptive MTP; re-measure on the current recipe.
{
match: { hw: "gb300", strategy: "low-latency" },
sglang_version: "d6ab04bdf1",
@@ -35,37 +36,86 @@ export const benchmarks = [
},
{
match: { hw: "gb300", strategy: "low-latency", kvDsaPair: "fp8-trtllm", quant: "fp8" },
- sglang_version: "d6ab04bdf1",
+ sglang_version: "b3dc0388ed",
latencyPercentile: "Mean",
speed: [
{
workload: {
dataset: "random",
- isl: 1024,
- osl: 256,
+ isl: 1000,
+ osl: 1000,
max_concurrency: 1,
- num_prompts: 8,
+ num_prompts: 80,
},
- ttft_ms: 211.25,
- tpot_ms: 3.65,
- tokens_per_sec_per_gpu: 279.53,
+ ttft_ms: 204.79,
+ tpot_ms: 2.0,
+ tokens_per_sec_per_gpu: 227.06,
},
{
workload: {
dataset: "random",
- isl: 1024,
- osl: 256,
+ isl: 1000,
+ osl: 1000,
max_concurrency: 16,
num_prompts: 80,
},
- ttft_ms: 609.9,
- tpot_ms: 6.25,
- tokens_per_sec_per_gpu: 2317.35,
+ ttft_ms: 367.48,
+ tpot_ms: 4.8,
+ tokens_per_sec_per_gpu: 1492.93,
+ },
+ {
+ workload: {
+ dataset: "random",
+ isl: 1000,
+ osl: 1000,
+ max_concurrency: 80,
+ num_prompts: 80,
+ },
+ ttft_ms: 1243.26,
+ tpot_ms: 6.28,
+ tokens_per_sec_per_gpu: 4487.92,
+ },
+ {
+ workload: {
+ dataset: "random",
+ isl: 8000,
+ osl: 1000,
+ max_concurrency: 1,
+ num_prompts: 80,
+ },
+ ttft_ms: 237.02,
+ tpot_ms: 1.95,
+ tokens_per_sec_per_gpu: 1029.64,
+ },
+ {
+ workload: {
+ dataset: "random",
+ isl: 8000,
+ osl: 1000,
+ max_concurrency: 16,
+ num_prompts: 80,
+ },
+ ttft_ms: 986.68,
+ tpot_ms: 5.16,
+ tokens_per_sec_per_gpu: 5550.31,
+ },
+ {
+ workload: {
+ dataset: "random",
+ isl: 8000,
+ osl: 1000,
+ max_concurrency: 80,
+ num_prompts: 80,
+ },
+ ttft_ms: 7085.4,
+ tpot_ms: 11.65,
+ tokens_per_sec_per_gpu: 8557.31,
},
],
notes:
- "The Low Latency recipe with FP8 KV + TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) on the current release-image tree (d6ab04bdf1), adaptive MTP 5/1/6 with SGLANG_SIMULATE_ACC_LEN=3 (accept 3.00): 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,853.88 aggregate output tok/s — 1.6% above the BF16 + TileLang Low Latency row, with mean TPOT 6.25 ms vs 6.43 ms. Draft and target full-graph capture succeeded for this combination. The speed rows were measured with the NEXTN spelling and --disable-shared-experts-fusion, which resolve to the same runtime path as the published command on this tree. Re-verified on the current release image (tree fe236ea6c3) within 1.5% on 4x GB300; the concurrency-1 entry (8 requests) comes from that re-run.",
+ "The published Low Latency command with FP8 KV + TRT-LLM DSA (TP4, EP1, flashinfer_trtllm MoE, fixed MTP 5/1/6 with natural acceptance) on 4x GB300, zai-org/GLM-5.3-Flash (eb9eb208eb0d) on upstream main b3dc0388ed with FlashInfer 0.6.18, measured in the 2026-09 GLM-5.3-Flash perf campaign (run 20260911_glm53flash_vs_vllm, lane sglang-mtp-trtllm-ep1). Each row is 80 ShareGPT-length random-token requests over streaming /v1/completions with ignore_eos, temperature 0, seed 42, prefix cache disabled, --context-length 16384, --max-running-requests 80, --cuda-graph-max-bs-decode 80 and --mem-fraction-static 0.90; chat is 1,000 input / 1,000 output tokens and summarization 8,000 / 1,000. TTFT and TPOT are per-run means; c16 and c80 are the median of two runs, c1 a single run. Aggregate output tok/s: chat 454.11 / 2,985.85 / 8,975.84 and summarization 457.62 / 2,466.80 / 3,803.25 at concurrency 1 / 16 / 80, at accept lengths of 5.98 / 5.98 / 4.7 (chat) and 6.0 / 6.0 / 4.5 (summarization). In the same campaign, fixed-depth MTP drafting measured 8,490.07 chat c80 tok/s against 3,664.53 for adaptive drafting on the TP4/EP4 deep_gemm command, and EP1 + flashinfer_trtllm then improved every cell over that fixed-depth command by 4.4-13.6%. The campaign's GSM8K natural-stop gate (all 1,319 problems, reasoning effort low, 8,192 max tokens, temperature 1.0 / top_p 0.95) ran these launch flags on eleven source candidates and every run stopped naturally on all 1,319 problems at 90.8-92.7%; that protocol differs from the thinking-mode GSM8K behind the accuracy figures on this page.",
},
+ // TODO: speed measured on the earlier TP4/EP4 deep_gemm command with adaptive MTP; re-measure on the current recipe.
{
match: { hw: "gb300", strategy: "low-latency", kvDsaPair: "fp8-trtllm", dcp: "4", quant: "fp8" },
sglang_version: "d6ab04bdf1",
@@ -87,6 +137,7 @@ export const benchmarks = [
notes:
"The Low Latency recipe with FP8 KV + TRT-LLM DSA and DCP4 (--dcp-size 4 --dcp-comm-backend a2a --dcp-replicate-q-proj) on 4x GB300, final weights (c5b82b63e37b) on the d6ab04bdf1 tree, adaptive MTP 5/1/6 with full decode graph: 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,680.61 aggregate output tok/s at a 3.937 accept length — about 10% below the non-DCP FP8 Low Latency row. TRT-LLM DSA DCP decode returns the LSE natively, so this arm needs no patch. Re-verified on the current release image (tree fe236ea6c3) within 1.3% on 4x GB300.",
},
+ // TODO: speed measured on the earlier TP4/EP4 deep_gemm command with adaptive MTP; re-measure on the current recipe.
{
match: { hw: "gb300", strategy: "low-latency", kvDsaPair: "bf16-tilelang", dcp: "4", quant: "fp8" },
sglang_version: "d6ab04bdf1",
@@ -108,6 +159,7 @@ export const benchmarks = [
notes:
"The Low Latency recipe with BF16 KV + TileLang DSA and DCP4 on 4x GB300, final weights (c5b82b63e37b) on the d6ab04bdf1 tree, adaptive MTP 5/1/6 with full decode graph: 80 random requests at 1,024 input / 256 output tokens and concurrency 16 produced 1,565.8 aggregate output tok/s at a 3.90 accept length. TileLang DSA DCP decode needs the LSE fix that ships in the current release image. Re-verified on the current release image (tree fe236ea6c3) within 1.3% on 4x GB300.",
},
+ // TODO: speed measured on the earlier TP4/EP4 deep_gemm command; re-measure on the current recipe.
{
match: { hw: "gb300", strategy: "high-throughput" },
sglang_version: "d6ab04bdf1",
@@ -154,6 +206,7 @@ export const benchmarks = [
notes:
"Measured on 4x GB300 (TP4/EP4) with the final weights (zai-org/GLM-5.3-Flash, c5b82b63e37b) on the current release-image tree (d6ab04bdf1), speculative decoding off, after two discarded warmups per row: 1,161.22 / 2,660.24 / 4,828.33 aggregate output tok/s at concurrency 16 / 64 / 256 (80 / 320 / 1,280 random requests at 1,024 input / 256 output tokens). The server ran exactly the published cell command. Throughput at 256 is still scaling but sublinear (prefill queueing). Accuracy is from the shared non-simulated full GSM8K gate: 97.50% with a 100% stop rate over all 1,319 problems. With HiCache L1+L2 (32 GB host tier, 16k prefill chunks) the same protocol measured 1,202.07 / 2,696.20 / 4,634.47 tok/s — within 4% of the non-HiCache rows; the random dataset has no prefix reuse, so L2 benefit was not exercised. Re-verified on the current release image (tree fe236ea6c3) within 1.5% on 4x GB300.",
},
+ // TODO: speed measured on the earlier TP4/EP4 deep_gemm command; re-measure on the current recipe.
{
match: { hw: "gb300", strategy: "high-throughput", kvDsaPair: "fp8-trtllm", quant: "fp8" },
sglang_version: "d6ab04bdf1",
@@ -200,6 +253,7 @@ export const benchmarks = [
notes:
"FP8 KV cache with TRT-LLM DSA on 4x GB300, final weights (c5b82b63e37b) on the current release-image tree (d6ab04bdf1), same protocol as the BF16 rows: 1,227.07 / 2,738.61 / 4,977.02 aggregate output tok/s at concurrency 16 / 64 / 256 — 2.9–5.7% above BF16 + TileLang across the curve, and the FP8 pool holds 12.6M tokens per rank vs 7.0M at BF16 (1.8x capacity at identical pool bytes). Accuracy is the full GSM8K gate on this variant: 97.35% vs 97.50% on BF16 KV, a 0.15-point gap inside sampling noise, with a 100% stop rate over all 1,319 problems. With HiCache L1+L2 (32 GB host tier, 16k prefill chunks) the same protocol measured 1,263.85 / 2,763.31 / 4,773.95 tok/s — within 5% of the non-HiCache rows; the random dataset has no prefix reuse, so L2 benefit was not exercised. Re-verified on the current release image (tree fe236ea6c3) within 1.5% on 4x GB300.",
},
+ // TODO: speed measured with adaptive MTP; re-measure with fixed MTP 5/1/6.
{
match: { hw: "gb300", strategy: "low-latency", quant: "nvfp4", kvDsaPair: "bf16-tilelang" },
sglang_version: "fe236ea6c3",
@@ -280,6 +334,7 @@ export const benchmarks = [
notes:
"RadixArk/GLM-5.3-Flash-NVFP4 with speculative decoding off — same checkpoint, image, and 4x GB300 measurement stack as the NVFP4 Low Latency row (ModelOpt 0.46.0 NVFP4 W4A4, abs-max, group size 16; MoE and dense MLPs in FP4, attention/router/MTP/embeddings BF16). Accuracy is a checkpoint-level result carried from that arm: GSM8K 97.14% over the full 1,319-example split x 4 seeds (per-seed range 96.89-97.42%, stop rate 99.85-100%) and AIME 2026 92.45% (30 problems x 16 repeats x 4 seeds, per-seed range 91.67-93.54%). Those runs used the NEXTN spelling of --speculative-algorithm on the adaptive-MTP arm, which resolves to the same runtime path as the published EAGLE command. Speed measured on 4x GB300 with the current release image (tree fe236ea6c3): 80 / 320 / 1,280 random requests at concurrency 16 / 64 / 256 (1,024 input / 256 output tokens) produced 1,352.86 / 3,291.36 / 5,919.49 aggregate output tok/s after two discarded warmups — at or above B300 parity, and above the FP8 gb300 High Throughput row at c256 (5,919.49 vs 4,828.33), as expected for W4A4.",
},
+ // TODO: speed measured with adaptive MTP; re-measure with fixed MTP 5/1/6.
{
match: { hw: "gb300", strategy: "low-latency", quant: "nvfp4", kvDsaPair: "fp8-trtllm" },
sglang_version: "fe236ea6c3",
@@ -358,6 +413,7 @@ export const benchmarks = [
notes:
"FP8 KV + TRT-LLM DSA pairing of the NVFP4 recipe with speculative decoding off — same TP4-only flashinfer_cutlass stack as the NVFP4 fp8-trtllm Low Latency row. Speed measured on 4x GB300 with the current release image (tree fe236ea6c3): 80 / 320 / 1,280 random requests at concurrency 16 / 64 / 256 (1,024 input / 256 output tokens) produced 1,439.25 / 3,428.58 / 6,150.46 aggregate output tok/s after two discarded warmups — at or above B300 parity, and above the FP8 gb300 High Throughput row at c256 (6,150.46 vs 4,977.02), as expected for W4A4.",
},
+ // TODO: speed measured with adaptive MTP; re-measure with fixed MTP 5/1/6.
{
match: { hw: "h100", strategy: "low-latency" },
sglang_version: "f040cc72e6",
@@ -413,6 +469,7 @@ export const benchmarks = [
notes:
"Full GSM8K (all 1,319 problems) on 8x H200 (TP8/EP8) with zai-org/GLM-5.3-Flash at f040cc72e6: 97.35% for the recommended selection; 97.19-97.57% across all 4 measured selections. Run with `sgl-eval run gsm8k --base-url http://localhost:30000/v1 --num-threads 32 --max-tokens 32768`; gsm8k's registered default leaves thinking off, so these are non-thinking numbers and are not directly comparable to the GB300 rows above. Accuracy only, no speed measurement.",
},
+ // TODO: speed measured on the earlier TP8/EP8 deep_gemm command with adaptive MTP; re-measure on the current recipe.
{
match: { hw: "b200", strategy: "low-latency" },
sglang_version: "f040cc72e6",
@@ -454,6 +511,7 @@ export const benchmarks = [
notes:
"Full GSM8K (all 1,319 problems) on 8x B200 (TP8/EP8) with zai-org/GLM-5.3-Flash at f040cc72e6: 97.27% for the recommended selection; 96.97-97.35% across all 8 measured selections. Run with `sgl-eval run gsm8k --base-url http://localhost:30000/v1 --num-threads 32 --max-tokens 32768`; gsm8k's registered default leaves thinking off, so these are non-thinking numbers and are not directly comparable to the GB300 rows above. Accuracy only, no speed measurement.",
},
+ // TODO: speed measured on the earlier TP8/EP8 deep_gemm command with adaptive MTP; re-measure on the current recipe.
{
match: { hw: "b300", strategy: "low-latency" },
sglang_version: "f040cc72e6",
diff --git a/docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx b/docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
index 0c6da4f62..36e199c5e 100644
--- a/docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
+++ b/docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
@@ -8,7 +8,7 @@ export const config = {
id: "strategy",
title: "Strategy",
options: [
- { id: "low-latency", label: "Low Latency", subtitle: "Adaptive MTP 5/1/6" },
+ { id: "low-latency", label: "Low Latency", subtitle: "MTP 5/1/6" },
{ id: "high-throughput", label: "High Throughput", subtitle: "Spec decode off" },
],
},
@@ -130,9 +130,9 @@ export const config = {
id: "4",
label: "DCP 4",
disabled: (s) => s.hw !== "gb300",
- disableReason: "DCP is validated only on 4x GB300 TP4/EP4 for now.",
+ disableReason: "DCP is validated only on 4x GB300 for now.",
flags: ["--dcp-size 4", "--dcp-comm-backend a2a", "--dcp-replicate-q-proj"],
- hints: ["Measured on 4x GB300 with both KV/DSA pairings, adaptive MTP 5/1/6, full decode graph."],
+ hints: ["Measured on 4x GB300 with both KV/DSA pairings and full decode graph on the earlier TP4/EP4 recipe; re-measurement on the current command is pending."],
},
],
},
@@ -158,8 +158,9 @@ export const config = {
benchmarkCommands: {
speed:
-`# Low Latency speed runs serve with SGLANG_SIMULATE_ACC_LEN=3 to pin the accept
-# length; that number is throughput evidence only. Never run accuracy against it.
+`# Low Latency speed rows whose notes mention SGLANG_SIMULATE_ACC_LEN=3 were served
+# with it to pin the accept length; that number is throughput evidence only. Never
+# run accuracy against it.
python3 -m sglang.bench_serving \\
--backend sglang \\
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
@@ -240,10 +241,10 @@ sgl-eval run gsm8k \\
disable: [
{
when: { strategy: ["low-latency"] },
- reason: "Low Latency uses adaptive MTP, which does not support DP-Attention.",
+ reason: "MTP speculative decoding with DP-Attention is not validated on this model.",
},
],
- disableReason: "Low Latency uses adaptive MTP, which does not support DP-Attention.",
+ disableReason: "MTP speculative decoding with DP-Attention is not validated on this model.",
},
],
},
@@ -282,7 +283,7 @@ sgl-eval run gsm8k \\
// ----- Card: "Speculative" -----
// The Deploy panel only picks speculation through the Strategy dim (Low
- // Latency = the checkpoint's adaptive MTP head, High Throughput = off).
+ // Latency = the checkpoint's MTP head, High Throughput = off).
// This card is the finer control, and it adds the one algorithm no cell
// ships: DFlash2, whose draft is a separate checkpoint.
//
@@ -295,18 +296,17 @@ sgl-eval run gsm8k \\
{ id: "off", label: "Off (greedy)" },
{
id: "eagle",
- label: "EAGLE / Adaptive MTP 5-1-6",
+ label: "EAGLE / MTP 5-1-6",
flags: [
"--speculative-algorithm EAGLE",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
],
disable: [
{
when: { dpAttnOn: [true] },
- reason: "Adaptive MTP does not support DP-Attention — the server falls back to a static draft depth and warns. Turn DP-Attention off in the Attention card above.",
+ reason: "MTP speculative decoding with DP-Attention is not validated on this model. Turn DP-Attention off in the Attention card above.",
},
],
},
@@ -346,27 +346,23 @@ sgl-eval run gsm8k \\
nnodes: 1,
verified: true,
verificationStatus: (s) =>
- s.bcg !== "off" ? "unverified" :
- ["bf16-tilelang", "fp8-trtllm"].includes(s.kvDsaPair) &&
- s.mmTransport === "auto" &&
- s.hicache === "off" &&
- ["off", "4"].includes(s.dcp)
- ? "verified"
+ s.bcg !== "off" || s.mmTransport !== "auto" || s.hicache !== "off" ? "unverified" :
+ s.kvDsaPair === "fp8-trtllm" && s.dcp === "off" ? "verified" :
+ ["bf16-tilelang", "fp8-trtllm"].includes(s.kvDsaPair) && ["off", "4"].includes(s.dcp)
+ ? "in-progress"
: "unverified",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
- "--ep-size 4",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -383,17 +379,16 @@ sgl-eval run gsm8k \\
s.mmTransport === "auto" &&
s.hicache === "off" &&
s.dcp === "off"
- ? "verified"
+ ? "in-progress"
: "unverified",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
- "--ep-size 4",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -415,7 +410,7 @@ sgl-eval run gsm8k \\
s.mmTransport === "auto" &&
s.hicache === "off" &&
s.dcp === "off"
- ? "verified"
+ ? "in-progress"
: "unverified",
env: [],
flags: [
@@ -430,7 +425,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
@@ -487,7 +481,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
@@ -533,7 +526,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
@@ -579,7 +571,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--mem-fraction-static 0.85",
@@ -608,6 +599,7 @@ sgl-eval run gsm8k \\
"--port {{PORT}}",
],
},
+ // Hopper keeps EP8 + deep_gemm: FlashInfer's trtllm-gen MoE kernels are SM100-only.
{
match: { hw: "h100", strategy: "low-latency", quant: "fp8" },
nnodes: 1,
@@ -615,7 +607,7 @@ sgl-eval run gsm8k \\
verificationStatus: (s) =>
s.bcg !== "off" ? "unverified" :
s.mmTransport === "auto" && s.hicache === "off"
- ? "verified"
+ ? "in-progress"
: "unverified",
env: [],
flags: [
@@ -631,7 +623,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -668,7 +659,7 @@ sgl-eval run gsm8k \\
verificationStatus: (s) =>
s.bcg !== "off" ? "unverified" :
s.mmTransport === "auto" && s.hicache === "off"
- ? "verified"
+ ? "in-progress"
: "unverified",
env: [],
flags: [
@@ -684,7 +675,6 @@ sgl-eval run gsm8k \\
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -718,21 +708,19 @@ sgl-eval run gsm8k \\
nnodes: 1,
verified: true,
verificationStatus: (s) =>
- s.bcg !== "off" ? "unverified" : (s.hicache === "off" ? "verified" : "unverified"),
+ s.bcg !== "off" ? "unverified" : (s.hicache === "off" ? "in-progress" : "unverified"),
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
- "--ep-size 8",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -745,16 +733,15 @@ sgl-eval run gsm8k \\
verified: true,
verificationStatus: (s) =>
s.bcg !== "off" ? "unverified" :
- ["off", "l2"].includes(s.hicache) ? "verified" : "unverified",
+ ["off", "l2"].includes(s.hicache) ? "in-progress" : "unverified",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
- "--ep-size 8",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -766,21 +753,19 @@ sgl-eval run gsm8k \\
nnodes: 1,
verified: true,
verificationStatus: (s) =>
- s.bcg !== "off" ? "unverified" : (s.hicache === "off" ? "verified" : "unverified"),
+ s.bcg !== "off" ? "unverified" : (s.hicache === "off" ? "in-progress" : "unverified"),
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
- "--ep-size 8",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -793,16 +778,15 @@ sgl-eval run gsm8k \\
verified: true,
verificationStatus: (s) =>
s.bcg !== "off" ? "unverified" :
- ["off", "l2"].includes(s.hicache) ? "verified" : "unverified",
+ ["off", "l2"].includes(s.hicache) ? "in-progress" : "unverified",
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 8",
- "--ep-size 8",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -817,16 +801,14 @@ sgl-eval run gsm8k \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
- "--ep-size 4",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 5",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 6",
- "--speculative-adaptive",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",
@@ -841,11 +823,10 @@ sgl-eval run gsm8k \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp-size 4",
- "--ep-size 4",
"--dsa-prefill-backend trtllm",
"--dsa-decode-backend trtllm",
"--kv-cache-dtype fp8_e4m3",
- "--moe-runner-backend deep_gemm",
+ "--moe-runner-backend flashinfer_trtllm",
"--reasoning-parser glm45",
"--tool-call-parser glm47",
"--host {{HOST_IP}}",