This commit is contained in:
zijiexia
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.3-Flash
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using NVIDIA CUDA and AMD ROCm recipes, with MTP and multimodal serving where validated."
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, and GB300, with MTP and multimodal serving."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -10,7 +10,7 @@ tag: NEW
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Use an SGLang build that includes GLM-5.3-Flash support. The AMD ROCm recipes also require [the ROCm engine changes in PR #36607](https://github.com/sgl-project/sglang/pull/36607) until they are available in a published SGLang image.
|
||||
Use an SGLang build that includes GLM-5.3-Flash support.
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:glm-5.3-flash
|
||||
@@ -25,7 +25,7 @@ Choose your hardware, then choose the operating point that matches your workload
|
||||
- **Low Latency** starts with adaptive MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
|
||||
- **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.
|
||||
|
||||
NVIDIA platforms expose both strategies. AMD ROCm currently exposes only the non-speculative High Throughput recipe because MTP has not been validated there. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
|
||||
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
|
||||
|
||||
The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to **Not Verified** without hiding the option.
|
||||
|
||||
@@ -69,7 +69,7 @@ GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a h
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>Precision</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights; FP8 KV cache by default on Blackwell, BF16 KV cache on H100, H200, and the AMD ROCm recipes</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights; FP8 KV cache by default on Blackwell, BF16 KV cache on H100 and H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Context</td>
|
||||
@@ -94,7 +94,7 @@ The deployment recipes use the checkpoint's generation configuration. Override s
|
||||
|
||||
Start with **Low Latency** for chat and agent workloads. Adaptive MTP changes the draft depth as acceptance changes, reducing unnecessary draft work when the server is busy. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value.
|
||||
|
||||
Strategy labels describe the workload goal. Both strategies stay available on NVIDIA GPUs; the AMD ROCm recipes expose only High Throughput until MTP speculative decoding is validated there.
|
||||
Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes.
|
||||
|
||||
### Change the speculative algorithm
|
||||
|
||||
@@ -104,7 +104,7 @@ The **Speculative** card in the Playground changes the algorithm without leaving
|
||||
- **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from.
|
||||
- **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. It needs a build that carries the GLM-5.3-Flash hidden-state capture from [PR #36708](https://github.com/sgl-project/sglang/pull/36708), which is merged into the [PR #36507](https://github.com/sgl-project/sglang/pull/36507) support branch (`xinyuan/glm-5.3-flash-support`) rather than into `main`, so the image pinned above is not enough on its own — pull that branch at its current head, or add #36708's commit on top of an older checkout. The draft repository is also access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point.
|
||||
|
||||
Neither algorithm runs with DP-Attention, and neither is available on the AMD ROCm recipes; the card disables the affected chips and names the reason.
|
||||
Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason.
|
||||
|
||||
### Size both memory pools
|
||||
|
||||
@@ -118,8 +118,6 @@ Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not o
|
||||
|
||||
On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination.
|
||||
|
||||
On AMD ROCm, use BF16 KV cache with TileLang DSA, set `SGLANG_USE_AITER=1`, keep the MoE runner on Triton, and disable CUDA graphs. The ROCm recipe uses TP8 on a single eight-GPU node. MI300X and MI325X both use gfx942, but the MI325X entry remains explicitly unverified because it is inferred from MI300X rather than measured directly. AMD validation covers text generation and GSM8K only; multimodal serving remains unverified.
|
||||
|
||||
### Decode context parallelism
|
||||
|
||||
Decode context parallelism (DCP) shards the KV across GPUs during decode to cut long-context latency. The **Context Parallelism** row offers DCP4, validated on 4x GB300 (TP4/EP4) with both KV pairings and adaptive MTP 5/1/6; it requires the current release image, which carries the TileLang LSE fix. Other platforms and attention backends are unvalidated, and draft-extend v2 is unsupported under DCP.
|
||||
|
||||
@@ -310,19 +310,4 @@ export const benchmarks = [
|
||||
{ match: { hw: "b200", strategy: "high-throughput", quant: "nvfp4" } },
|
||||
{ match: { hw: "b300", strategy: "low-latency", quant: "nvfp4" } },
|
||||
{ match: { hw: "b300", strategy: "high-throughput", quant: "nvfp4" } },
|
||||
{
|
||||
match: { hw: "mi300x", strategy: "high-throughput" },
|
||||
sglang_version: "9e692c9216",
|
||||
accuracy: { gsm8k_pct: 97.35 },
|
||||
notes:
|
||||
"Accuracy-only validation on 8x MI300X (gfx942, TP8) with zai-org/GLM-5.3-Flash revision 3f1971b7b5f7a528c9c4ef6212c8785298a8c24a, SGLang PR #36607 head 9e692c9216c3b5e5c443fecf6b995700eb68d2e4 (validated source manifest 2c240e0e01d5fdf04acc485ebfa25f8a1793ba45fb07f165eecedfba7ec1db80), and lmsysorg/sglang:v0.5.18-rocm720-mi30x with the PR source mounted over the image tree. Full GSM8K scored 1,284/1,319 with a 100% stop rate and zero request errors, empty generations, or truncations. No throughput or latency benchmark was run.",
|
||||
},
|
||||
{ match: { hw: "mi325x", strategy: "high-throughput" } },
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "high-throughput" },
|
||||
sglang_version: "9e692c9216",
|
||||
accuracy: { gsm8k_pct: 97.65 },
|
||||
notes:
|
||||
"Accuracy-only validation on 8x MI355X (gfx950, TP8) with zai-org/GLM-5.3-Flash revision 3f1971b7b5f7a528c9c4ef6212c8785298a8c24a, SGLang PR #36607 head 9e692c9216c3b5e5c443fecf6b995700eb68d2e4 (validated source manifest 2c240e0e01d5fdf04acc485ebfa25f8a1793ba45fb07f165eecedfba7ec1db80), and lmsysorg/sglang:v0.5.18-rocm720-mi35x with the PR source mounted over the image tree. Full GSM8K scored 1,288/1,319 with a 100% stop rate and zero request errors, empty generations, or truncations. No throughput or latency benchmark was run.",
|
||||
},
|
||||
];
|
||||
|
||||
@@ -1,23 +1,14 @@
|
||||
export const config = {
|
||||
modelName: "GLM-5.3-Flash",
|
||||
|
||||
supportedHardware: [
|
||||
"gb300", "h100", "h200", "b200", "b300", "gb200",
|
||||
"mi300x", "mi325x", "mi355x",
|
||||
],
|
||||
supportedHardware: ["gb300", "h100", "h200", "b200", "b300", "gb200"],
|
||||
|
||||
matchDims: [
|
||||
{
|
||||
id: "strategy",
|
||||
title: "Strategy",
|
||||
options: [
|
||||
{
|
||||
id: "low-latency",
|
||||
label: "Low Latency",
|
||||
subtitle: "Adaptive MTP 5/1/6",
|
||||
disabled: (s) => ["mi300x", "mi325x", "mi355x"].includes(s.hw),
|
||||
disableReason: "MTP speculative decoding has not been validated for GLM-5.3-Flash on AMD ROCm; use the non-speculative High Throughput recipe.",
|
||||
},
|
||||
{ id: "low-latency", label: "Low Latency", subtitle: "Adaptive MTP 5/1/6" },
|
||||
{ id: "high-throughput", label: "High Throughput", subtitle: "Spec decode off" },
|
||||
],
|
||||
},
|
||||
@@ -30,16 +21,14 @@ export const config = {
|
||||
id: "nvfp4",
|
||||
label: "NVFP4",
|
||||
disabled: (s) => !["gb300", "gb200", "b200", "b300"].includes(s.hw),
|
||||
disableReason: "The NVFP4 W4A4 kernels are Blackwell-only; Hopper and AMD ROCm cannot serve this checkpoint.",
|
||||
disableReason: "The NVFP4 W4A4 kernels are Blackwell-only; Hopper cannot serve this checkpoint.",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
|
||||
isRecommendedSelection(s) {
|
||||
const pairing = ["h100", "h200", "mi300x", "mi325x", "mi355x"].includes(s.hw)
|
||||
? "bf16-tilelang"
|
||||
: "fp8-trtllm";
|
||||
const pairing = ["h100", "h200"].includes(s.hw) ? "bf16-tilelang" : "fp8-trtllm";
|
||||
return (
|
||||
s.kvDsaPair === pairing &&
|
||||
s.mmTransport === "auto" &&
|
||||
@@ -57,8 +46,8 @@ export const config = {
|
||||
{
|
||||
id: "fp8-trtllm",
|
||||
label: "FP8 + TRT-LLM",
|
||||
disabled: (s) => ["h100", "h200", "mi300x", "mi325x", "mi355x"].includes(s.hw),
|
||||
disableReason: "This recipe uses BF16 KV cache with TileLang DSA on Hopper and AMD ROCm GPUs.",
|
||||
disabled: (s) => ["h100", "h200"].includes(s.hw),
|
||||
disableReason: "FP8 KV cache with TRT-LLM DSA is not supported on Hopper GPUs.",
|
||||
stripPrefixes: ["--kv-cache-dtype", "--dsa-prefill-backend", "--dsa-decode-backend"],
|
||||
flags: [
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
@@ -186,9 +175,8 @@ sgl-eval run gsm8k \\
|
||||
["aime2026_pct", "AIME 2026", "%"],
|
||||
],
|
||||
|
||||
// Support is not in a public sglang release yet. NVIDIA uses the
|
||||
// purpose-built CUDA 13 image. AMD validation used these ROCm 7.2 images
|
||||
// with the GLM-5.3 ROCm engine branch mounted over the image source tree.
|
||||
// Support is not in a public sglang release yet, so the nightly images do
|
||||
// not work; every NVIDIA lane uses the purpose-built CUDA 13 image.
|
||||
dockerImages: {
|
||||
gb300: "lmsysorg/sglang:glm-5.3-flash",
|
||||
h100: "lmsysorg/sglang:glm-5.3-flash",
|
||||
@@ -196,9 +184,6 @@ sgl-eval run gsm8k \\
|
||||
b200: "lmsysorg/sglang:glm-5.3-flash",
|
||||
b300: "lmsysorg/sglang:glm-5.3-flash",
|
||||
gb200: "lmsysorg/sglang:glm-5.3-flash",
|
||||
mi300x: "lmsysorg/sglang:v0.5.18-rocm720-mi30x",
|
||||
mi325x: "lmsysorg/sglang:v0.5.18-rocm720-mi30x",
|
||||
mi355x: "lmsysorg/sglang:v0.5.18-rocm720-mi35x",
|
||||
},
|
||||
|
||||
github: {
|
||||
@@ -256,8 +241,6 @@ sgl-eval run gsm8k \\
|
||||
id: "deep_gemm",
|
||||
label: "DeepGemm",
|
||||
flags: ["--moe-runner-backend deep_gemm"],
|
||||
disabled: (s) => ["mi300x", "mi325x", "mi355x"].includes(s.hw),
|
||||
disableReason: "The validated AMD ROCm recipe uses the Triton MoE runner.",
|
||||
},
|
||||
],
|
||||
},
|
||||
@@ -310,10 +293,6 @@ sgl-eval run gsm8k \\
|
||||
when: { dpAttnOn: [true] },
|
||||
reason: "Adaptive MTP does not support DP-Attention — the server falls back to a static draft depth and warns. Turn DP-Attention off in the Attention card above.",
|
||||
},
|
||||
{
|
||||
when: { hw: ["mi300x", "mi325x", "mi355x"] },
|
||||
reason: "MTP speculative decoding has not been validated for GLM-5.3-Flash on AMD ROCm; the Strategy row disables Low Latency there for the same reason.",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -339,10 +318,6 @@ sgl-eval run gsm8k \\
|
||||
when: { dpAttnOn: [true] },
|
||||
reason: "DFLASH speculative decoding does not support DP-Attention — the server rejects the combination at startup. Turn DP-Attention off in the Attention card above.",
|
||||
},
|
||||
{
|
||||
when: { hw: ["mi300x", "mi325x", "mi355x"] },
|
||||
reason: "DFLASH speculative decoding only supports CUDA and NPU devices; the server rejects it on ROCm at startup.",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
@@ -858,71 +833,5 @@ sgl-eval run gsm8k \\
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// AMD ROCm — one non-speculative TP8 operating point. The explicit BF16
|
||||
// KV and TileLang DSA flags match the resolved defaults observed in the
|
||||
// validation server logs. AITER remains enabled for the ROCm kernel paths,
|
||||
// while Triton owns the MoE runner. CUDA graphs stay disabled because that
|
||||
// is the architecture-gated configuration used for correctness validation.
|
||||
{
|
||||
match: { hw: "mi300x", strategy: "high-throughput", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
verified: true,
|
||||
env: ["SGLANG_USE_AITER=1"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--trust-remote-code",
|
||||
"--disable-cuda-graph",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--kv-cache-dtype bfloat16",
|
||||
"--moe-runner-backend triton",
|
||||
"--reasoning-parser glm45",
|
||||
"--tool-call-parser glm47",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi325x", strategy: "high-throughput", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
warn: "This MI325X recipe is inferred from the validated MI300X gfx942 path. It has not been measured directly on MI325X.",
|
||||
env: ["SGLANG_USE_AITER=1"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--trust-remote-code",
|
||||
"--disable-cuda-graph",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--kv-cache-dtype bfloat16",
|
||||
"--moe-runner-backend triton",
|
||||
"--reasoning-parser glm45",
|
||||
"--tool-call-parser glm47",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "high-throughput", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
verified: true,
|
||||
env: ["SGLANG_USE_AITER=1"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp-size 8",
|
||||
"--trust-remote-code",
|
||||
"--disable-cuda-graph",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
"--kv-cache-dtype bfloat16",
|
||||
"--moe-runner-backend triton",
|
||||
"--reasoning-parser glm45",
|
||||
"--tool-call-parser glm47",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user