diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index 2c5cf3751..118ba83e6 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -34,7 +34,7 @@ the full contract): | `quantizations` | `{id, label}[]` | 3rd-dim option list. | | `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. | | `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. | -| `cells` | `{match, verified?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). | +| `cells` | `{match, verified?, verificationStatus?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). | | `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). | | `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. | | `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. | @@ -114,6 +114,17 @@ Each cell describes one verified (or auto-estimated) launch recipe. - `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`, `strategy`, `nodes`. The engine looks up cells by tuple equality. +- `verified` is the badge baseline (`true` → green **Verified**, absent → + yellow **Not Verified**). `verificationStatus` overrides it with a third + state — `"verified" | "in-progress" | "unverified"` — for a recipe whose + verification round is OPEN rather than absent. It may also be a FUNCTION of + the selection, which is how a cell reports a per-pick state: e.g. + `verificationStatus: (sel) => sel.spec === "dflash" ? "in-progress" : + "verified"` marks one speculative option as still being validated while the + cell's other picks stay Verified. An unrecognized string falls back to + `unverified`, so a typo can never render as a green badge — and + `check_cookbook_configs.mjs` probes the function over every reachable + selection, so a typo or a crash fails the check instead of reaching the page. - `env` and `flags` are FLAT literals. The engine does NOT expand fragments, aliases, or templates — it consumes them verbatim (only `{{PLACEHOLDER}}` substitutions happen at render time). diff --git a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl index da644495f..51d87188a 100644 --- a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl +++ b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl @@ -335,6 +335,9 @@ sgl-eval run gsm8k \\ // ==== NVIDIA Blackwell + FP4 (single node) ==== { match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" }, + // For a cell whose verification round is open, use `verificationStatus` + // instead ("verified" | "in-progress" | "unverified", or a function of + // the selection for a per-pick state) — authoring-reference §2.2. verified: true, // EXAMPLE — set false / replace with your verified recipe env: [], flags: [ diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx index 4d2a705bc..c62e95fb5 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx @@ -20,6 +20,9 @@ For all methods and hardware platforms, see the [official SGLang installation gu pip install --upgrade pip pip install uv uv pip install sglang + +# For the DFLASH2 cells only — DFlash2 selector support is newer than the +# latest release, so install from source (Method 2) instead of the line above. ``` Then run the **Python** output of the command panel below in that environment. @@ -30,6 +33,10 @@ Then run the **Python** output of the command panel below in that environment. ```bash Command docker pull lmsysorg/sglang:qwen38-27b + +# For the DFLASH2 cells only — that tag predates DFlash2 selector support, +# so pull a nightly built from main instead: +# docker pull lmsysorg/sglang:dev-nightly-0820 ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. @@ -38,6 +45,17 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d + +**DFLASH2 needs a build that tracks main.** DFlash2 landed in +[#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + NVFP4 +— the quantized `lm_head` path — in +[#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer than +the pinned `lmsysorg/sglang:qwen38-27b` tag and than the latest PyPI release; a +build without #35496 fails on the NVFP4 cells at boot with `requires a dense +FP16/BF16/FP32 target lm_head`. Every other recipe on this page — no +speculation, MTP, DSpark — runs on the pinned tag as written. + + Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point. @@ -56,19 +74,34 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca + + **Picking DFLASH2 requires SGLang built from main.** DFlash2 landed in + [#35371](https://github.com/sgl-project/sglang/pull/35371), and DFlash2 + + NVFP4 — the quantized `lm_head` path — in + [#35496](https://github.com/sgl-project/sglang/pull/35496). Both are newer + than the `lmsysorg/sglang:qwen38-27b` tag pinned in + [Install SGLang](#install) and than the latest PyPI release; on NVFP4 a build + without #35496 fails at boot with `requires a dense FP16/BF16/FP32 target + lm_head`. Every other Speculative Decoding pick — None, EAGLE, DSPARK — runs + on the pinned tag as written. + + The RTX 5090 and RTX PRO 6000 cells above — including every Speculative Decoding / Serving Strategy / SSM dtype combination — were validated at - ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on - NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the - remaining platforms the DFLASH2 pick is offered but not yet exercised, and - the composed command carries a `# DFLASH2 on this platform: final - verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus - DFLASH2, but to a weaker standard: each was confirmed to **boot and - serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or - acceptance-length numbers taken. The remaining platforms' recipes carry their - original validation, which covers the default overlay picks (plus MTP on - GB300); non-default overlay picks there are valid but unmeasured. + ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on + NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark + cells cover the same combination set minus DFLASH2, but to a weaker standard: + each was confirmed to **boot and serve** at ISL 8192 / OSL 1024, + concurrency 1, with no throughput or acceptance-length numbers taken. The + H200 and GB300 recipes carry their original validation, which covers the + default overlay picks (plus MTP on GB300); non-default overlay picks there + are valid but unmeasured. + + DFLASH2 has not been exercised on H200, DGX Spark or GB300: selecting it + there switches the cell's badge to **Final Verification In Progress** until + that round lands. The other picks on those platforms keep the validation + described above. ### Mamba ratio calculator @@ -234,8 +267,8 @@ checkpoint's calibration scales automatically. is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300 - the pick is offered with a final-verification-in-progress hint line - above the command. The + those cells carry the **Final Verification In Progress** + badge. The RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer `--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept diff --git a/docs/scripts/check_cookbook_configs.mjs b/docs/scripts/check_cookbook_configs.mjs index aa3a7b8bb..35b610687 100755 --- a/docs/scripts/check_cookbook_configs.mjs +++ b/docs/scripts/check_cookbook_configs.mjs @@ -169,6 +169,20 @@ for (const path of walk(CONFIGS)) { } } }; + // A cell may report its badge per selection (`verificationStatus` as a + // function of sel), so it has to survive the same space the predicates do — + // it renders on every pick, and an unrecognized return silently downgrades + // the badge to "Not Verified" rather than erroring in the browser. + const VERIFY_STATES = ["verified", "in-progress", "unverified"]; + for (const [i, cell] of (config.cells || []).entries()) { + if (typeof cell.verificationStatus !== "function") continue; + probe((sel) => { + const out = cell.verificationStatus(sel); + if (out !== undefined && !VERIFY_STATES.includes(out)) { + throw new Error(`returned ${JSON.stringify(out)}, expected one of [${VERIFY_STATES}]`); + } + }, `cells[${i}].verificationStatus`); + } for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) { if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`); for (const opt of (dim.options || [])) { diff --git a/docs/src/snippets/_deployment.jsx b/docs/src/snippets/_deployment.jsx index 86493840e..d36a7c6ba 100644 --- a/docs/src/snippets/_deployment.jsx +++ b/docs/src/snippets/_deployment.jsx @@ -38,7 +38,10 @@ // `verified` is the boolean badge baseline. // `verificationStatus` overrides it with a third state — // "verified" | "in-progress" | "unverified" — for a recipe -// whose verification round is open rather than absent. +// whose verification round is open rather than absent. It +// may also be a function of the selection, for a cell whose +// verification depends on an overlay pick (e.g. one +// speculative option still being validated). // modelNames HF slug lookup, `hw|variant|quant`, `variant|quant`, // `hw|quant`, `quant`, `hw`, then `default` // placeholders {{KEY}} → {target: 'command'|'curl', label, default?} @@ -443,8 +446,14 @@ export const Deployment = ({ config, benchmarks }) => { typeof v === "string" ? (VERIFY_LABEL[v] ? v : "unverified") : (v ? "verified" : "unverified"); - const cellVerifyStatus = (c) => - c ? verifyStatusOf(c.verificationStatus ?? c.verified) : "unverified"; + const cellVerifyStatus = (c, sel) => { + if (!c) return "unverified"; + const v = + typeof c.verificationStatus === "function" + ? c.verificationStatus(sel) + : c.verificationStatus; + return verifyStatusOf(v ?? c.verified); + }; // Two kinds of selector row: // match dims participate in cell lookup (cell.match[dim] === sel[dim]) @@ -1240,7 +1249,7 @@ export const Deployment = ({ config, benchmarks }) => { // ==== 5. Derived values ==== const s = makeStyles(isDark); const cell = findCell(config.cells, sel); - const verifyStatus = cellVerifyStatus(cell); + const verifyStatus = cellVerifyStatus(cell, sel); // Pin the calculator-computed ratio into the rendered command (before the // host/port tail); cells themselves stay ratio-free. const cellWithRatio = (() => { diff --git a/docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx b/docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx index c681499db..482de8a17 100644 --- a/docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx +++ b/docs/src/snippets/configs/Qwen/qwen3.8-27b.jsx @@ -122,17 +122,12 @@ export const config = { // selector projects through the target lm_head — including // quantized heads — so it runs on the NVFP4 checkpoint too. // Validated on the SM120 pair (NVFP4 measured end to end; the - // RTX PRO 6000 BF16/FP8 cells boot-and-serve). On the other - // platforms the pick is offered with the in-progress hint below - // — the recipe composes from validated cells but has not been - // exercised there yet. + // RTX PRO 6000 BF16/FP8 cells boot-and-serve). The platforms where + // it has not been exercised carry verificationStatus "in-progress" + // on their cells. disabled: (sel) => sel.hw === "rtx5090" && sel.quant !== "nvfp4", disableReason: "On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights", - hints: (sel) => - ["rtx6000", "rtx5090"].includes(sel.hw) - ? [] - : ["DFLASH2 on this platform: final verification in progress"], // 5090: mem-fraction re-pins like DSPARK's, and fp32 additionally // re-pins the ratio — the balanced L=9216 value leaves the state // pool one slot short at every serviceable mem-fraction (see the @@ -405,6 +400,10 @@ export const config = { // runnable, but not a recipe this page ships. match: { hw: "h200", variant: "default", quant: "fp8", nodes: "single" }, verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -424,6 +423,10 @@ export const config = { // H200, BF16 reference checkpoint (~54GB of weights). match: { hw: "h200", variant: "default", quant: "bf16", nodes: "single" }, verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -545,6 +548,10 @@ export const config = { // exercises the 4-bit `lm_head` this checkpoint quantizes, with no shape // error. verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -563,6 +570,10 @@ export const config = { match: { hw: "dgx-spark", variant: "default", quant: "fp8", nodes: "single" }, // All 12 overlay combinations served on GB10. verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -582,6 +593,10 @@ export const config = { // All 12 overlay combinations served on GB10. Heaviest checkpoint, so // it holds the sweep's tightest cell: DSPARK + float32 + extra_buffer. verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -603,6 +618,10 @@ export const config = { { match: { hw: "gb300", variant: "default", quant: "nvfp4", nodes: "single" }, verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -619,6 +638,10 @@ export const config = { { match: { hw: "gb300", variant: "default", quant: "fp8", nodes: "single" }, verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code", @@ -635,6 +658,10 @@ export const config = { { match: { hw: "gb300", variant: "default", quant: "bf16", nodes: "single" }, verified: true, + // DFLASH2 has not been exercised on this platform; every other overlay + // pick keeps this cell's original validation. + verificationStatus: (sel) => + sel.spec === "dflash" ? "in-progress" : "verified", env: [], flags: [ "--trust-remote-code",