[docs] Tell Qwen3.8-27B DFLASH2 users to build from main (#35753)

This commit is contained in:
Jimmy Shong
2026-08-20 23:34:44 +00:00
committed by GitHub
parent 779e593bd1
commit 1a138e13b9
6 changed files with 122 additions and 25 deletions
@@ -34,7 +34,7 @@ the full contract):
| `quantizations` | `{id, label}[]` | 3rd-dim option list. | | `quantizations` | `{id, label}[]` | 3rd-dim option list. |
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. | | `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. | | `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
| `cells` | `{match, verified?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). | | `cells` | `{match, verified?, verificationStatus?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). |
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). | | `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. | | `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. | | `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
@@ -114,6 +114,17 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`, - `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`,
`strategy`, `nodes`. The engine looks up cells by tuple equality. `strategy`, `nodes`. The engine looks up cells by tuple equality.
- `verified` is the badge baseline (`true` → green **Verified**, absent →
yellow **Not Verified**). `verificationStatus` overrides it with a third
state — `"verified" | "in-progress" | "unverified"` — for a recipe whose
verification round is OPEN rather than absent. It may also be a FUNCTION of
the selection, which is how a cell reports a per-pick state: e.g.
`verificationStatus: (sel) => sel.spec === "dflash" ? "in-progress" :
"verified"` marks one speculative option as still being validated while the
cell's other picks stay Verified. An unrecognized string falls back to
`unverified`, so a typo can never render as a green badge — and
`check_cookbook_configs.mjs` probes the function over every reachable
selection, so a typo or a crash fails the check instead of reaching the page.
- `env` and `flags` are FLAT literals. The engine does NOT expand - `env` and `flags` are FLAT literals. The engine does NOT expand
fragments, aliases, or templates — it consumes them verbatim fragments, aliases, or templates — it consumes them verbatim
(only `{{PLACEHOLDER}}` substitutions happen at render time). (only `{{PLACEHOLDER}}` substitutions happen at render time).
@@ -335,6 +335,9 @@ sgl-eval run gsm8k \\
// ==== NVIDIA Blackwell + FP4 (single node) ==== // ==== NVIDIA Blackwell + FP4 (single node) ====
{ {
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" }, match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
// For a cell whose verification round is open, use `verificationStatus`
// instead ("verified" | "in-progress" | "unverified", or a function of
// the selection for a per-pick state) — authoring-reference §2.2.
verified: true, // EXAMPLE — set false / replace with your verified recipe verified: true, // EXAMPLE — set false / replace with your verified recipe
env: [], env: [],
flags: [ flags: [
@@ -20,6 +20,9 @@ For all methods and hardware platforms, see the [official SGLang installation gu
pip install --upgrade pip pip install --upgrade pip
pip install uv pip install uv
uv pip install sglang uv pip install sglang
# For the DFLASH2 cells only — DFlash2 selector support is newer than the
# latest release, so install from source (Method 2) instead of the line above.
``` ```
Then run the **Python** output of the command panel below in that environment. Then run the **Python** output of the command panel below in that environment.
@@ -30,6 +33,10 @@ Then run the **Python** output of the command panel below in that environment.
```bash Command ```bash Command
docker pull lmsysorg/sglang:qwen38-27b docker pull lmsysorg/sglang:qwen38-27b
# For the DFLASH2 cells only — that tag predates DFlash2 selector support,
# so pull a nightly built from main instead:
# docker pull lmsysorg/sglang:dev-nightly-0820
``` ```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -38,6 +45,17 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
</Tabs> </Tabs>
<Warning>
**DFLASH2 needs a build that tracks main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + NVFP4
— the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer than
the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI release; a
build without #35496 fails on the NVFP4 cells at boot with `requires a dense
FP16/BF16/FP32 target lm_head`. Every other recipe on this page — no
speculation, MTP, DSpark — runs on the pinned tag as written.
</Warning>
</Accordion> </Accordion>
Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point. Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point.
@@ -56,19 +74,34 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} /> <Deployment config={config} />
<Warning>
**Picking DFLASH2 requires SGLang built from main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 +
NVFP4 — the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer
than the `lmsysorg/sglang:qwen38-27b` tag pinned in
[Install SGLang](#install) and than the latest PyPI release; on NVFP4 a build
without #35496 fails at boot with `requires a dense FP16/BF16/FP32 target
lm_head`. Every other Speculative Decoding pick — None, EAGLE, DSPARK — runs
on the pinned tag as written.
</Warning>
<Note> <Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on
NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark
remaining platforms the DFLASH2 pick is offered but not yet exercised, and cells cover the same combination set minus DFLASH2, but to a weaker standard:
the composed command carries a `# DFLASH2 on this platform: final each was confirmed to **boot and serve** at ISL 8192 / OSL 1024,
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus concurrency 1, with no throughput or acceptance-length numbers taken. The
DFLASH2, but to a weaker standard: each was confirmed to **boot and H200 and GB300 recipes carry their original validation, which covers the
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or default overlay picks (plus MTP on GB300); non-default overlay picks there
acceptance-length numbers taken. The remaining platforms' recipes carry their are valid but unmeasured.
original validation, which covers the default overlay picks (plus MTP on
GB300); non-default overlay picks there are valid but unmeasured. DFLASH2 has not been exercised on H200, DGX Spark or GB300: selecting it
there switches the cell's badge to **Final Verification In Progress** until
that round lands. The other picks on those platforms keep the validation
described above.
</Note> </Note>
### Mamba ratio calculator ### Mamba ratio calculator
@@ -234,8 +267,8 @@ checkpoint's calibration scales automatically.
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300 RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
the pick is offered with a final-verification-in-progress hint line those cells carry the **Final Verification In Progress**
above the command. The badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured `--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
+14
View File
@@ -169,6 +169,20 @@ for (const path of walk(CONFIGS)) {
} }
} }
}; };
// A cell may report its badge per selection (`verificationStatus` as a
// function of sel), so it has to survive the same space the predicates do —
// it renders on every pick, and an unrecognized return silently downgrades
// the badge to "Not Verified" rather than erroring in the browser.
const VERIFY_STATES = ["verified", "in-progress", "unverified"];
for (const [i, cell] of (config.cells || []).entries()) {
if (typeof cell.verificationStatus !== "function") continue;
probe((sel) => {
const out = cell.verificationStatus(sel);
if (out !== undefined && !VERIFY_STATES.includes(out)) {
throw new Error(`returned ${JSON.stringify(out)}, expected one of [${VERIFY_STATES}]`);
}
}, `cells[${i}].verificationStatus`);
}
for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) { for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) {
if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`); if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`);
for (const opt of (dim.options || [])) { for (const opt of (dim.options || [])) {
+13 -4
View File
@@ -38,7 +38,10 @@
// `verified` is the boolean badge baseline. // `verified` is the boolean badge baseline.
// `verificationStatus` overrides it with a third state — // `verificationStatus` overrides it with a third state —
// "verified" | "in-progress" | "unverified" — for a recipe // "verified" | "in-progress" | "unverified" — for a recipe
// whose verification round is open rather than absent. // whose verification round is open rather than absent. It
// may also be a function of the selection, for a cell whose
// verification depends on an overlay pick (e.g. one
// speculative option still being validated).
// modelNames HF slug lookup, `hw|variant|quant`, `variant|quant`, // modelNames HF slug lookup, `hw|variant|quant`, `variant|quant`,
// `hw|quant`, `quant`, `hw`, then `default` // `hw|quant`, `quant`, `hw`, then `default`
// placeholders {{KEY}} → {target: 'command'|'curl', label, default?} // placeholders {{KEY}} → {target: 'command'|'curl', label, default?}
@@ -443,8 +446,14 @@ export const Deployment = ({ config, benchmarks }) => {
typeof v === "string" typeof v === "string"
? (VERIFY_LABEL[v] ? v : "unverified") ? (VERIFY_LABEL[v] ? v : "unverified")
: (v ? "verified" : "unverified"); : (v ? "verified" : "unverified");
const cellVerifyStatus = (c) => const cellVerifyStatus = (c, sel) => {
c ? verifyStatusOf(c.verificationStatus ?? c.verified) : "unverified"; if (!c) return "unverified";
const v =
typeof c.verificationStatus === "function"
? c.verificationStatus(sel)
: c.verificationStatus;
return verifyStatusOf(v ?? c.verified);
};
// Two kinds of selector row: // Two kinds of selector row:
// match dims participate in cell lookup (cell.match[dim] === sel[dim]) // match dims participate in cell lookup (cell.match[dim] === sel[dim])
@@ -1240,7 +1249,7 @@ export const Deployment = ({ config, benchmarks }) => {
// ==== 5. Derived values ==== // ==== 5. Derived values ====
const s = makeStyles(isDark); const s = makeStyles(isDark);
const cell = findCell(config.cells, sel); const cell = findCell(config.cells, sel);
const verifyStatus = cellVerifyStatus(cell); const verifyStatus = cellVerifyStatus(cell, sel);
// Pin the calculator-computed ratio into the rendered command (before the // Pin the calculator-computed ratio into the rendered command (before the
// host/port tail); cells themselves stay ratio-free. // host/port tail); cells themselves stay ratio-free.
const cellWithRatio = (() => { const cellWithRatio = (() => {
+35 -8
View File
@@ -122,17 +122,12 @@ export const config = {
// selector projects through the target lm_head — including // selector projects through the target lm_head — including
// quantized heads — so it runs on the NVFP4 checkpoint too. // quantized heads — so it runs on the NVFP4 checkpoint too.
// Validated on the SM120 pair (NVFP4 measured end to end; the // Validated on the SM120 pair (NVFP4 measured end to end; the
// RTX PRO 6000 BF16/FP8 cells boot-and-serve). On the other // RTX PRO 6000 BF16/FP8 cells boot-and-serve). The platforms where
// platforms the pick is offered with the in-progress hint below // it has not been exercised carry verificationStatus "in-progress"
// — the recipe composes from validated cells but has not been // on their cells.
// exercised there yet.
disabled: (sel) => sel.hw === "rtx5090" && sel.quant !== "nvfp4", disabled: (sel) => sel.hw === "rtx5090" && sel.quant !== "nvfp4",
disableReason: disableReason:
"On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights", "On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights",
hints: (sel) =>
["rtx6000", "rtx5090"].includes(sel.hw)
? []
: ["DFLASH2 on this platform: final verification in progress"],
// 5090: mem-fraction re-pins like DSPARK's, and fp32 additionally // 5090: mem-fraction re-pins like DSPARK's, and fp32 additionally
// re-pins the ratio — the balanced L=9216 value leaves the state // re-pins the ratio — the balanced L=9216 value leaves the state
// pool one slot short at every serviceable mem-fraction (see the // pool one slot short at every serviceable mem-fraction (see the
@@ -405,6 +400,10 @@ export const config = {
// runnable, but not a recipe this page ships. // runnable, but not a recipe this page ships.
match: { hw: "h200", variant: "default", quant: "fp8", nodes: "single" }, match: { hw: "h200", variant: "default", quant: "fp8", nodes: "single" },
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -424,6 +423,10 @@ export const config = {
// H200, BF16 reference checkpoint (~54GB of weights). // H200, BF16 reference checkpoint (~54GB of weights).
match: { hw: "h200", variant: "default", quant: "bf16", nodes: "single" }, match: { hw: "h200", variant: "default", quant: "bf16", nodes: "single" },
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -545,6 +548,10 @@ export const config = {
// exercises the 4-bit `lm_head` this checkpoint quantizes, with no shape // exercises the 4-bit `lm_head` this checkpoint quantizes, with no shape
// error. // error.
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -563,6 +570,10 @@ export const config = {
match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" }, match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" },
// All 12 overlay combinations served on GB10. // All 12 overlay combinations served on GB10.
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -582,6 +593,10 @@ export const config = {
// All 12 overlay combinations served on GB10. Heaviest checkpoint, so // All 12 overlay combinations served on GB10. Heaviest checkpoint, so
// it holds the sweep's tightest cell: DSPARK + float32 + extra_buffer. // it holds the sweep's tightest cell: DSPARK + float32 + extra_buffer.
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -603,6 +618,10 @@ export const config = {
{ {
match: { hw: "gb300", variant: "default", quant: "nvfp4", nodes: "single" }, match: { hw: "gb300", variant: "default", quant: "nvfp4", nodes: "single" },
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -619,6 +638,10 @@ export const config = {
{ {
match: { hw: "gb300", variant: "default", quant: "fp8", nodes: "single" }, match: { hw: "gb300", variant: "default", quant: "fp8", nodes: "single" },
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",
@@ -635,6 +658,10 @@ export const config = {
{ {
match: { hw: "gb300", variant: "default", quant: "bf16", nodes: "single" }, match: { hw: "gb300", variant: "default", quant: "bf16", nodes: "single" },
verified: true, verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [], env: [],
flags: [ flags: [
"--trust-remote-code", "--trust-remote-code",