minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[docs] Tell Qwen3.8-27B DFLASH2 users to build from main (#35753)

Browse Source
This commit is contained in:
Jimmy Shong
2026-08-20 23:34:44 +00:00
committed by GitHub
parent 779e593bd1
commit 1a138e13b9
6 changed files with 122 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/cookbook-add-model/references/authoring-reference.md
+12 -1
View File
@@ -34,7 +34,7 @@ the full contract):
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
| `cells` | `{match, verified?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). |
| `cells` | `{match, verified?, verificationStatus?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). |
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
@@ -114,6 +114,17 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`,
`strategy`, `nodes`. The engine looks up cells by tuple equality.
- `verified` is the badge baseline (`true` → green **Verified**, absent →
yellow **Not Verified**). `verificationStatus` overrides it with a third
state — `"verified" | "in-progress" | "unverified"` — for a recipe whose
verification round is OPEN rather than absent. It may also be a FUNCTION of
the selection, which is how a cell reports a per-pick state: e.g.
`verificationStatus: (sel) => sel.spec === "dflash" ? "in-progress" :
"verified"` marks one speculative option as still being validated while the
cell's other picks stay Verified. An unrecognized string falls back to
`unverified`, so a typo can never render as a green badge — and
`check_cookbook_configs.mjs` probes the function over every reachable
selection, so a typo or a crash fails the check instead of reaching the page.
- `env` and `flags` are FLAT literals. The engine does NOT expand
fragments, aliases, or templates — it consumes them verbatim
(only `{{PLACEHOLDER}}` substitutions happen at render time).
.claude/skills/cookbook-add-model/templates/config.jsx.tmpl
+3
View File
@@ -335,6 +335,9 @@ sgl-eval run gsm8k \\
// ==== NVIDIA Blackwell + FP4 (single node) ====
{
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
// For a cell whose verification round is open, use `verificationStatus`
// instead ("verified" | "in-progress" | "unverified", or a function of
// the selection for a per-pick state) — authoring-reference §2.2.
verified: true, // EXAMPLE — set false / replace with your verified recipe
env: [],
flags: [
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+45 -12
View File
@@ -20,6 +20,9 @@ For all methods and hardware platforms, see the [official SGLang installation gu
pip install --upgrade pip
pip install uv
uv pip install sglang
# For the DFLASH2 cells only — DFlash2 selector support is newer than the
# latest release, so install from source (Method 2) instead of the line above.
```
Then run the **Python** output of the command panel below in that environment.
@@ -30,6 +33,10 @@ Then run the **Python** output of the command panel below in that environment.
```bash Command
docker pull lmsysorg/sglang:qwen38-27b
# For the DFLASH2 cells only — that tag predates DFlash2 selector support,
# so pull a nightly built from main instead:
# docker pull lmsysorg/sglang:dev-nightly-0820
```
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
@@ -38,6 +45,17 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
</Tabs>
<Warning>
**DFLASH2 needs a build that tracks main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + NVFP4
— the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer than
the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI release; a
build without #35496 fails on the NVFP4 cells at boot with `requires a dense
FP16/BF16/FP32 target lm_head`. Every other recipe on this page — no
speculation, MTP, DSpark — runs on the pinned tag as written.
</Warning>
</Accordion>
Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point.
@@ -56,19 +74,34 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} />
<Warning>
**Picking DFLASH2 requires SGLang built from main.** DFlash2 landed in
[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 +
NVFP4 — the quantized `lm_head` path — in
[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer
than the `lmsysorg/sglang:qwen38-27b` tag pinned in
[Install SGLang](#install) and than the latest PyPI release; on NVFP4 a build
without #35496 fails at boot with `requires a dense FP16/BF16/FP32 target
lm_head`. Every other Speculative Decoding pick — None, EAGLE, DSPARK — runs
on the pinned tag as written.
</Warning>
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on
NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the
remaining platforms the DFLASH2 pick is offered but not yet exercised, and
the composed command carries a `# DFLASH2 on this platform: final
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus
DFLASH2, but to a weaker standard: each was confirmed to **boot and
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
acceptance-length numbers taken. The remaining platforms' recipes carry their
original validation, which covers the default overlay picks (plus MTP on
GB300); non-default overlay picks there are valid but unmeasured.
ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on
NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark
cells cover the same combination set minus DFLASH2, but to a weaker standard:
each was confirmed to **boot and serve** at ISL 8192 / OSL 1024,
concurrency 1, with no throughput or acceptance-length numbers taken. The
H200 and GB300 recipes carry their original validation, which covers the
default overlay picks (plus MTP on GB300); non-default overlay picks there
are valid but unmeasured.
DFLASH2 has not been exercised on H200, DGX Spark or GB300: selecting it
there switches the cell's badge to **Final Verification In Progress** until
that round lands. The other picks on those platforms keep the validation
described above.
</Note>
### Mamba ratio calculator
@@ -234,8 +267,8 @@ checkpoint's calibration scales automatically.
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
the pick is offered with a final-verification-in-progress hint line
above the command. The
those cells carry the **Final Verification In Progress**
badge. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
docs/scripts/check_cookbook_configs.mjs
+14
View File
@@ -169,6 +169,20 @@ for (const path of walk(CONFIGS)) {
}
}
};
// A cell may report its badge per selection (`verificationStatus` as a
// function of sel), so it has to survive the same space the predicates do —
// it renders on every pick, and an unrecognized return silently downgrades
// the badge to "Not Verified" rather than erroring in the browser.
const VERIFY_STATES = ["verified", "in-progress", "unverified"];
for (const [i, cell] of (config.cells || []).entries()) {
if (typeof cell.verificationStatus !== "function") continue;
probe((sel) => {
const out = cell.verificationStatus(sel);
if (out !== undefined && !VERIFY_STATES.includes(out)) {
throw new Error(`returned ${JSON.stringify(out)}, expected one of [${VERIFY_STATES}]`);
}
}, `cells[${i}].verificationStatus`);
}
for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) {
if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`);
for (const opt of (dim.options || [])) {
docs/src/snippets/_deployment.jsx
+13 -4
View File
@@ -38,7 +38,10 @@
// `verified` is the boolean badge baseline.
// `verificationStatus` overrides it with a third state —
// "verified" | "in-progress" | "unverified" — for a recipe
// whose verification round is open rather than absent.
// whose verification round is open rather than absent. It
// may also be a function of the selection, for a cell whose
// verification depends on an overlay pick (e.g. one
// speculative option still being validated).
// modelNames HF slug lookup, `hw|variant|quant`, `variant|quant`,
// `hw|quant`, `quant`, `hw`, then `default`
// placeholders {{KEY}} → {target: 'command'|'curl', label, default?}
@@ -443,8 +446,14 @@ export const Deployment = ({ config, benchmarks }) => {
typeof v === "string"
? (VERIFY_LABEL[v] ? v : "unverified")
: (v ? "verified" : "unverified");
const cellVerifyStatus = (c) =>
c ? verifyStatusOf(c.verificationStatus ?? c.verified) : "unverified";
const cellVerifyStatus = (c, sel) => {
if (!c) return "unverified";
const v =
typeof c.verificationStatus === "function"
? c.verificationStatus(sel)
: c.verificationStatus;
return verifyStatusOf(v ?? c.verified);
};
// Two kinds of selector row:
// match dims participate in cell lookup (cell.match[dim] === sel[dim])
@@ -1240,7 +1249,7 @@ export const Deployment = ({ config, benchmarks }) => {
// ==== 5. Derived values ====
const s = makeStyles(isDark);
const cell = findCell(config.cells, sel);
const verifyStatus = cellVerifyStatus(cell);
const verifyStatus = cellVerifyStatus(cell, sel);
// Pin the calculator-computed ratio into the rendered command (before the
// host/port tail); cells themselves stay ratio-free.
const cellWithRatio = (() => {
docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx
+35 -8
View File
@@ -122,17 +122,12 @@ export const config = {
// selector projects through the target lm_head — including
// quantized heads — so it runs on the NVFP4 checkpoint too.
// Validated on the SM120 pair (NVFP4 measured end to end; the
// RTX PRO 6000 BF16/FP8 cells boot-and-serve). On the other
// platforms the pick is offered with the in-progress hint below
// — the recipe composes from validated cells but has not been
// exercised there yet.
// RTX PRO 6000 BF16/FP8 cells boot-and-serve). The platforms where
// it has not been exercised carry verificationStatus "in-progress"
// on their cells.
disabled: (sel) => sel.hw === "rtx5090" && sel.quant !== "nvfp4",
disableReason:
"On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights",
hints: (sel) =>
["rtx6000", "rtx5090"].includes(sel.hw)
? []
: ["DFLASH2 on this platform: final verification in progress"],
// 5090: mem-fraction re-pins like DSPARK's, and fp32 additionally
// re-pins the ratio — the balanced L=9216 value leaves the state
// pool one slot short at every serviceable mem-fraction (see the
@@ -405,6 +400,10 @@ export const config = {
// runnable, but not a recipe this page ships.
match: { hw: "h200", variant: "default", quant: "fp8", nodes: "single" },
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -424,6 +423,10 @@ export const config = {
// H200, BF16 reference checkpoint (~54GB of weights).
match: { hw: "h200", variant: "default", quant: "bf16", nodes: "single" },
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -545,6 +548,10 @@ export const config = {
// exercises the 4-bit `lm_head` this checkpoint quantizes, with no shape
// error.
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -563,6 +570,10 @@ export const config = {
match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" },
// All 12 overlay combinations served on GB10.
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -582,6 +593,10 @@ export const config = {
// All 12 overlay combinations served on GB10. Heaviest checkpoint, so
// it holds the sweep's tightest cell: DSPARK + float32 + extra_buffer.
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -603,6 +618,10 @@ export const config = {
{
match: { hw: "gb300", variant: "default", quant: "nvfp4", nodes: "single" },
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -619,6 +638,10 @@ export const config = {
{
match: { hw: "gb300", variant: "default", quant: "fp8", nodes: "single" },
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",
@@ -635,6 +658,10 @@ export const config = {
{
match: { hw: "gb300", variant: "default", quant: "bf16", nodes: "single" },
verified: true,
// DFLASH2 has not been exercised on this platform; every other overlay
// pick keeps this cell's original validation.
verificationStatus: (sel) =>
sel.spec === "dflash" ? "in-progress" : "verified",
env: [],
flags: [
"--trust-remote-code",