From 29ac249be19fc733ae58739cc12dce81dae76ee4 Mon Sep 17 00:00:00 2001 From: zijiexia <37504505+zijiexia@users.noreply.github.com> Date: Fri, 12 Jun 2026 18:31:52 -0700 Subject: [PATCH] docs: add cookbook-migrate-model skill from the Qwen3.5 pilot (#27845) Co-authored-by: Claude Fable 5 --- .claude/skills/cookbook-add-model/SKILL.md | 5 + .../references/authoring-reference.md | 23 +- .../references/engine-axis.md | 4 + .../references/mintlify-authoring.md | 9 +- .../templates/config.jsx.tmpl | 14 +- .../templates/page.mdx.tmpl | 18 +- .../skills/cookbook-migrate-model/SKILL.md | 219 ++++++++++++++++ .../references/dimension-mapping.md | 240 ++++++++++++++++++ .claude/skills/cookbook-review-pr/SKILL.md | 69 ++++- 9 files changed, 579 insertions(+), 22 deletions(-) create mode 100644 .claude/skills/cookbook-migrate-model/SKILL.md create mode 100644 .claude/skills/cookbook-migrate-model/references/dimension-mapping.md diff --git a/.claude/skills/cookbook-add-model/SKILL.md b/.claude/skills/cookbook-add-model/SKILL.md index 85193479f..4b8da4ed5 100644 --- a/.claude/skills/cookbook-add-model/SKILL.md +++ b/.claude/skills/cookbook-add-model/SKILL.md @@ -6,6 +6,11 @@ disable-model-invocation: true # Add a model to the SGLang Cookbook +> Migrating an **existing legacy-template page** (one that imports a monolithic +> `…/autoregressive/-deployment.jsx` generator)? Use the +> `cookbook-migrate-model` skill instead — same target format, but the legacy +> page (not the user) is the source of truth. + The cookbook is **config-driven**: two shared engines contain NO model-specific code — `docs_new/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and `_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**: diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index 770a063c0..6554812b8 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -32,7 +32,7 @@ the full contract): | `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. | | `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. | | `quantizations` | `{id, label}[]` | 3rd-dim option list. | -| `strategies` | `{id, label}[]` | 4th-dim option list. Common ids: `low-latency`, `balanced`, `high-throughput`. | +| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. | | `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. | | `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. | | `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). | @@ -91,6 +91,16 @@ Each cell describes one verified (or auto-estimated) launch recipe. flags, then tuning knobs, with `--host` / `--port` last. The playground engine assumes this ordering when inserting overrides (its anchors target `--model-path` / `--tp` / etc., and inserts before the `--host` tail). +- Accuracy-degrading flags don't belong in cells by default: a cell's + output quality should be exactly what its quantization chip declares. + Runtime quant below the checkpoint's precision (e.g. MegaMoE **W4A4** — + DSv4 gates it behind the Playground's `megamoeQuant` opt-in) and lossy + KV-cache dtypes (`--kv-cache-dtype fp8_e4m3` over a higher-precision-KV + checkpoint) default to Playground opt-ins or §2-tips material. If the + model's recipe genuinely needs one in a cell, **flag it to the user and + get explicit confirmation** — never ship it silently. (Migrations are the + sanctioned exception: a flag baked into the legacy recipe's default + command keeps verbatim — see the migrate skill.) **Cells are denormalized on purpose** — common flags repeat across cells. This makes each cell self-contained and easy to verify. When sweeping a @@ -103,9 +113,14 @@ automatically. ## 2.3 Configure `playgroundFeatures` (optional) -The Playground widget is opt-in per axis. Add only the axes that make sense -for this model. Recognised axis keys and their schemas (full reference in -the `_playground.jsx` header): +The Playground is **opt-out, not opt-in**: every cookbook ships the general +axes by default — `attention` (TP/CP/DP-Attn), `moe` (backend + EP, for MoE +models), `parsers`, `speculative`, `pdDisagg`, `hicache` — then adds +model-specific axes (e.g. MegaMoE for DeepSeek-V4) and deletes ONLY the axes +this model genuinely cannot use (e.g. `hisparse` on non-DSA models, `moe` on a +pure-dense model). Knobs that don't apply to a subset of variants/hw get +`disable` + `disableReason`, not removal. Recognised axis keys and their +schemas (full reference in the `_playground.jsx` header): | Axis key | Widget | Use when | |---|---|---| diff --git a/.claude/skills/cookbook-add-model/references/engine-axis.md b/.claude/skills/cookbook-add-model/references/engine-axis.md index 8d82c656b..92747e189 100644 --- a/.claude/skills/cookbook-add-model/references/engine-axis.md +++ b/.claude/skills/cookbook-add-model/references/engine-axis.md @@ -7,6 +7,10 @@ cookbook is data-only and never needs this. The current 7 built-in axes (MegaMoE is not its own axis — it lives inside `moe` as a backend option + a `megamoeQuant` sub-select.) Only add a new axis if a real cookbook needs it and the feature does not fit any existing axis. Touches `_playground.jsx` only. +One routine trigger: the `cookbook-migrate-model` skill requires every legacy +control to survive as a dimension or a Playground axis — a legacy feature no +built-in axis covers (e.g. Nemotron3's "KV Cache DType" radio) lands here +first, as its own engine PR ahead of the migration PR. For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md). diff --git a/.claude/skills/cookbook-add-model/references/mintlify-authoring.md b/.claude/skills/cookbook-add-model/references/mintlify-authoring.md index 76a62a419..56d96a63a 100644 --- a/.claude/skills/cookbook-add-model/references/mintlify-authoring.md +++ b/.claude/skills/cookbook-add-model/references/mintlify-authoring.md @@ -61,9 +61,12 @@ table is a live reference. - **Tool-call follow-up on thinking models**: the final assistant turn may put text in `reasoning_content` instead of (or with) `content` — print both so the output isn't a misleading `None`. -- **Every runnable block** is immediately followed by `**Output Example:**` + a - ` ```text Output ` block with **real** server output (verbatim, not paraphrased). - `Pending update...` is acceptable only with the user's explicit acknowledgement. +- **§3 commands and outputs are collapsible (required)**: every runnable example + lives in an `` and its **real** server output + (verbatim, not paraphrased) in an immediately following + `` — match the DeepSeek-V4 §3 pattern. No + inline `**Output Example:**` headings / bare blocks. `Pending update...` is + acceptable only with the user's explicit acknowledgement. - **Do not hardcode sampling params** (`temperature`, `top_p`) in sample code — SGLang uses `generation_config.json` defaults. Listing "Recommended generation" in §1 is fine. - Format raw API objects (`ChatCompletionMessage(...)`) into readable Reasoning / diff --git a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl index 9f6e49f1a..05d70ffd7 100644 --- a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl +++ b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl @@ -48,6 +48,12 @@ export const config = { { id: "fp8", label: "FP8" }, { id: "fp4", label: "FP4" }, ], + // 4th dim. The count follows the model's operating points: 1 recipe → a + // single "balanced"; 2 → low-latency + high-throughput; 3 → the full trio + // (the ideal). Per-combination: a single-recipe combination (e.g. a CPU + // platform) parks under its semantically honest tier — no slant → balanced; + // the page's list is the union and the engine greys unused chips. Never + // invent a recipe just to fill chips. strategies: [ { id: "low-latency", label: "Low-Latency" }, { id: "balanced", label: "Balanced" }, @@ -158,7 +164,10 @@ sgl-eval run gsm8k \\ cookbookModel: "__HF_ORG__/__MODEL_SLUG__", }, - // Opt-in per axis. DELETE any axis your model doesn't expose (don't leave a stub). + // Opt-OUT per axis: general axes (attention / moe-for-MoE / parsers / + // speculative / pdDisagg / hicache) ship on every cookbook — DELETE only the + // axes this model genuinely cannot use (e.g. hisparse on non-DSA models, moe + // on pure-dense models); prefer disable+disableReason for variant/hw subsets. playgroundFeatures: { // ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP @@ -312,6 +321,9 @@ sgl-eval run gsm8k \\ // EXAMPLE cells — one per hardware family to show the shape. REPLACE each with // your model's verified recipe, or DELETE families you don't support. `match` // MUST have exactly the 5 keys; env/flags are flat literals. + // Accuracy-degrading flags (W4A4-style runtime quant, lossy --kv-cache-dtype) + // default to Playground/tips — putting one in a cell needs explicit user + // confirmation (authoring-reference §2.2). cells: [ // ==== NVIDIA Blackwell + FP4 (single node) ==== { diff --git a/.claude/skills/cookbook-add-model/templates/page.mdx.tmpl b/.claude/skills/cookbook-add-model/templates/page.mdx.tmpl index 9711f9e3b..74afbb02a 100644 --- a/.claude/skills/cookbook-add-model/templates/page.mdx.tmpl +++ b/.claude/skills/cookbook-add-model/templates/page.mdx.tmpl @@ -103,8 +103,9 @@ import { Playground } from "/src/snippets/_playground.jsx"; ## 3. Advanced Usage -{/* Keep only the subsections that apply. Each runnable block is followed by an - **Output Example:** + a ```text Output block with REAL server output. */} +{/* Keep only the subsections that apply. Commands and outputs in this section are + COLLAPSIBLE (required — match DeepSeek-V4 §3): each runnable example lives in an + , its REAL server output in a following . */} ### 3.1 Reasoning @@ -114,6 +115,8 @@ Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser** answer → `content`). If your parser emits inline `...` tags inside `content`, parse the tags from `content` instead. */} + + ```python Example from openai import OpenAI @@ -128,18 +131,23 @@ print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` -**Output Example:** + + + ```text Output TODO: paste real server output here. ``` + + ### 3.2 Tool Calling Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. -{/* TODO: tool-calling example + **Output Example:**. On thinking-mode models the - follow-up may put text in `reasoning_content`; print both that and `content`. */} +{/* TODO: tool-calling example in an + an . + On thinking-mode models the follow-up may put text in `reasoning_content`; + print both that and `content`. */} ### 3.3 HiCache (Hierarchical KV Caching) diff --git a/.claude/skills/cookbook-migrate-model/SKILL.md b/.claude/skills/cookbook-migrate-model/SKILL.md new file mode 100644 index 000000000..882123760 --- /dev/null +++ b/.claude/skills/cookbook-migrate-model/SKILL.md @@ -0,0 +1,219 @@ +--- +name: cookbook-migrate-model +description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs_new/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model . +--- + +# Cookbook Migrate Model + +Convert one legacy cookbook page to the config-driven format, faithfully. The +legacy page — its generator widget and its measured benchmark blocks — is the +**single source of truth**. You are transcribing it into the new data model, +not improving it. + +Reuses the `cookbook-add-model` skill's assets (read them on demand): +- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl` +- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract) +- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules) + +Migration-specific references in this skill: +- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848). + +The round's per-model inventory (scope, batch order, quirks, measured-data +survey) is tracked by the migration maintainer outside the repo — expect it in +your dispatch prompt, or ask for it. + +## Hard rules (non-negotiable) + +1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings + are copied verbatim from the legacy page — even when today's defaults differ + (e.g. `SGLANG_ENABLE_SPEC_V2=1` is now default; keep it anyway). The recipe + that was verified is the recipe as written. Allowed normalizations are ONLY + the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang + serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated + `--speculative-algo`→`--speculative-algorithm`, + `--expert-parallel-size`→`--ep`). **Accuracy-degrading flags** + (`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a + deterministic rule — enforced in migration, no asking: offered as a + legacy **selectable option** → never select it (cells mirror the + accuracy-safe side), and the option **survives as a Playground axis** — + an existing one where it fits, else add one via a separate prior engine + PR (rule 4); a legacy choice never degrades to a tips mention. Baked + into the recipe's **default/unconditional command** → keep it verbatim + (the recipe was measured with it, and fp8 KV halves KV memory — + stripping could OOM it). See dimension-mapping.md §2 caveats. +2. **Never invent versions or numbers.** Benchmark numbers only from the + legacy page's measured blocks. **Speed measurements migrate ONLY when the + legacy page pins an exact, reproducible build** (a release tag or a commit + hash) — drifting strings like "main branch" are no version anchor: drop the + speed numbers AND the entry's `sglang_version`, keep accuracy (far less + build-sensitive) and `benchmarkCommands` so ⚡Reproduce guides + re-measurement against a pinned release; confirm ambiguous strings (e.g. + "0.5.8+") with the maintainer. When kept, `sglang_version` is the legacy + page's string verbatim. Docker tags only the ones the legacy page pinned + (unmapped hw falls back to `:dev`). +3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the + legacy page has concrete measured data for that exact 5-dim combo AND + (b) the cell's flags equal the deployment command used for that measurement + — modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser + flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every + cell (Playground-only feature; when the measured run had them on, say so in + the benchmarks file header). When the measured command diverges from the + generator default, **the verified cell follows the measured command**; the + generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) — + including combos that look memory-infeasible; keep them verbatim and list + them in the PR body for the re-verification track. +4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not + change in a migration PR. If the model needs an engine capability (a new + axis, accuracy labels, …), that is a separate prior PR. The every-feature + rule (step 2) triggers this routinely: a legacy control no built-in axis + covers means an engine-axis PR lands first. +5. **`github.cookbookModel` must be set** (`/`, e.g. + `qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels + as deepseek-v4. The issue template itself needs NO edits (free-form input). +6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` / + `high-throughput` ONLY on a signal present in the legacy source (an + explicit performance toggle, a named recipe, or prose stating the + operating point); **no signal → `balanced`**. Never derive a slant from + your own hardware intuition — re-tiering on measured evidence is the + hardware owner's follow-up PR, not part of a migration + (dimension-mapping.md §4). + +## Workflow (one model = one PR) + +### 1. Inventory the legacy assets +- Read the legacy generator (`docs_new/src/snippets/autoregressive/-deployment.jsx`) + end-to-end: every option dimension (radio vs checkbox vs dynamic), every + gate/SUPPORT matrix, the full emitted command per reachable combo (env + prefixes, `# Error` pseudo-commands included). +- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not + upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real + Output Examples verbatim); §5 benchmark blocks → transcribe each measured + block: deploy command used, bench command (dataset/isl/osl/num-prompts/ + concurrency), Mean TTFT/TPOT, output tok/s, hardware, version string. +- Inbound-anchor sweep: `grep -rn "" docs_new/ --include='*.mdx'` — + find links/`#fragments` into this page (`mint broken-links` does NOT check + fragments). Fix referrers or add `` shims in the same PR. +- Check the maintainer-provided inventory notes for this model's known quirks — + but treat them (and the §4 family table) as a **survey snapshot**: re-verify + every dimension against the live legacy files before mapping. Pages keep + receiving updates (precedent: Kimi-K2.6's live generator has a speculative + toggle the 2026-06-10 survey notes lack). + +### 2. Design the 5-dim mapping +Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key +decision — which legacy toggle becomes the `strategies` dimension: a toggle +that **changes other parts of the command** (TP, mem) must (the Playground +can't do coupled changes), and so does a toggle the legacy page itself labels +with operating-point words — e.g. a `dpattention` radio whose options are +subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) — +even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure +flag add). Any other toggle that only adds/removes its own flags becomes a +Playground axis with the flags baked into cells when the legacy default was +ON — EXCEPT parsers: +`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are +Playground-only (DSv4 convention). **Every legacy control survives as an +interactive control** — a dimension or a Playground axis, never a tips-only +mention. A feature none of the built-in axes covers (Nemotron3's "KV Cache +DType" radio is the precedent) still lands in the Playground: add the axis +via a separate PRIOR engine PR (engine-axis.md), keeping the migration PR +itself data-only (hard rule 4). The strategy count follows the page's +operating points: **one recipe → a single `balanced`; two → `low-latency` + +`high-throughput`; three → the full trio (the ideal)**. The tiers apply per +(hw × variant × quant) combination — a single-recipe combination on a +multi-strategy page parks under its semantically honest tier (no +latency/throughput slant → `balanced`; the page's list is the union). When the +legacy toggle is MTP / speculative decoding, the direction is a deterministic +default — apply without asking: **MTP on → `low-latency`, MTP off → +`high-throughput`** (reversed only with maintainer confirmation). Tier +placement is signal-driven (hard rule 6). Never invent a recipe just to fill +strategy chips (see dimension-mapping.md §4). Record the outcome as a +**strategy mapping table** for the PR body — one row per group of +combinations sharing the same legacy signal (e.g. "all GPU combos: MTP +toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a +one-line rationale each; don't enumerate 60 identical rows. The table is what +hardware owners sign off on at review. + +### 3. Generate the config (codegen, then audit) +- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node + script that enumerates combos and emits the `cells:[...]` literal (output + must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the + verified-cell override in the script. See the pilot scripts embedded in the + worked example of dimension-mapping.md §5. +- **Independent equivalence audit (required):** extract the ORIGINAL generator + from git (`git show main:` — NOT `HEAD:`, the migration branch deletes + the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for + every combo, and diff token-by-token against the new cells. Expected deltas + only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the + engine-injected multi-node trio, the §2 alias rewrites (the entrypoint + rewrite doesn't appear in cell tokens — cells hold flags only; the audit + script normalizes it on the legacy side), and the intentional verified-cell + override. Paste the PASS count + the audit script in the PR body + (collapsed `
`). +- Hand-author the non-cells fields per authoring-reference.md. Structural + self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/ + --dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every + `supportedHardware` id has ≥1 cell. + +### 4. Benchmarks file +One entry per measured block only (cells without entries already render +"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` = +output tok/s ÷ (tp × nnodes); TTFT/TPOT take the Mean rows; put the workload's +`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever +the benchmarks carry accuracy data** — the engine ships no default eval set +(#27842), so missing labels means the accuracy rows silently don't render; +extra context (sample counts, suites that don't fit) goes in the entry's +`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop +entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users. + +### 5. Rewrite the MDX +From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh +SEO `description` (top-level — delete any legacy `metatags.description`), **no +`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion +carries the legacy install content + pinned images. Keep the template's +DSv4-style strategy bullets — serving semantics first (single-user chat / +typical multi-user / batch throughput), trimmed to the strategies the page +ships, plus at most a one-line note on what each strategy changes on this +model; do NOT rewrite them as toggle-/migration-centric explanations. +Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands → +⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity — +fold their unique flags into §2 tips). Invocation examples + real outputs carry +over verbatim, but **wrapped in Accordions** — §3 commands and outputs are +collapsible (required, DeepSeek-V4 pattern): code in an +``, output in a following +``; legacy pages kept them inline. + +### 6. Delete the legacy generator +Remove `docs_new/src/snippets/autoregressive/-deployment.jsx` and its +import. `grep -rn "-deployment" docs_new/` must return nothing (config +provenance comments must not name the deleted path). Site wiring needs **no +changes**: docs.json path/title unchanged, vendor card + logo already exist. + +### 7. Validate +- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens). +- `cd docs_new && mint validate && mint broken-links` (pre-existing breaks on + main are not yours — say so in the PR). +- `mint dev` browser smoke: initial selection = the verified cell (first in + `cells[]`) with green badge; multi-node cells show the injected trio + + header; AMD cells show env prefixes; Docker mode wraps with the pinned image + and passes cell env as `--env`; condition-hidden combos grey out; benchmark + card values; NO parser flags in any Deploy command; Playground parser + toggles ADD the parser flags (green additions) while spec toggles strike + the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most + ONE programmatic click per evaluation and wait for React to settle — + multiple clicks in one synchronous script batch and read stale DOM. +- Token-level audit from step 3 passes. + +### 8. PR + review +One PR per model. PR body: migration framing, verified policy applied, the +strategy mapping table (step 2), the audit PASS count + script, any +inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr ` and fix findings. +FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches — +a fork-headed PR is perfectly fine but renders no preview; a maintainer can +re-push the branch in-repo if a preview is wanted for review. + +### 9. Keep this skill current +Any new convention, engine behavior, or pitfall you discover while migrating +(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed +back into this skill — same PR if it's skill-file-only, or an immediate +follow-up commit on the skill's branch/PR. The next agent runs on what's +written here, not on your session's context. diff --git a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md new file mode 100644 index 000000000..27b31758d --- /dev/null +++ b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md @@ -0,0 +1,240 @@ +# Legacy → config-driven dimension mapping + +Loaded on demand by the `cookbook-migrate-model` skill. How to translate a +legacy generator's option space into the 5-dim matrix + Playground axes. +Field schemas live in `../../cookbook-add-model/references/authoring-reference.md`; +this file is about the *mapping decisions*. + +## 1. Legacy control → new home + +| Legacy control | New home | Rule | +|---|---|---| +| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). | +| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). | +| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. | +| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). | +| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **No fitting built-in axis ≠ drop the feature** — EVERY legacy control survives as an interactive control (a dimension or a Playground axis), never a tips-only mention. When none of the built-in axes fits (Nemotron3's "KV Cache DType" radio is the precedent), add the axis via the engine-axis flow: a separate PRIOR engine PR, then the migration config declares it (hard rule 4). | +| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. | +| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending); Qwen3.6 / Qwen3-Coder-Next carry the same knob, so their migrations need the axis in place first. | + +## 2. Command rewrite table (the ONLY allowed normalizations) + +| Legacy | New | +|---|---| +| `python(3) -m sglang.launch_server` | (engine emits `sglang serve`; cells hold flags only) | +| `--model X` / `--model-path X` | `--model-path {{MODEL_NAME}}` + `modelNames` key | +| `--tp-size N` | `--tp N` | +| `--speculative-algo X` (abbreviated) | `--speculative-algorithm X` — the Playground spec axis strips/derives by the full first token only; an abbreviated alias would survive toggles and double up | +| `--expert-parallel-size N` | `--ep N` — the Playground EP knob recognizes/strips only `--ep`; the long form would survive toggles and double up | +| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell | +| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment | +| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) | +| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. | + +Caveats discovered in the pilot: +- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy + `--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob + can recognize/strip it. +- `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class); + standard-IB H100 multi-node needs none. +- `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped + (`:dev` fallback) with a "install from source" tip. +- **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style + runtime quant) — deterministic rule, enforced in migration without + asking: + - offered as a legacy **selectable option/toggle** → never select it; + cells mirror the accuracy-safe side (even if the legacy default was the + lossy side). The option itself **must survive as a Playground axis** — the user's choice may not degrade to a + tips mention. Use the existing axis when one fits (DSv4 gates W4A4 + behind `megamoeQuant`); when the playground lacks one — e.g. + Nemotron3-Ultra's "KV Cache DType" radio (None default / fp8_e4m3 / + bf16) has no kv-cache axis today — **add the axis** via the engine-axis + flow: a separate PRIOR engine PR (hard rule 4, like accuracyLabels + #27842), then the migration config declares it; + - baked into the recipe's **unconditional/default command** → keep it + verbatim. The legacy measurements ran with it, and fp8 KV halves KV + memory — stripping could OOM the recipe. Expect this pattern: legacy + AMD recipes routinely append `--kv-cache-dtype fp8_e4m3` ("for memory + efficiency"), and GLM-5's NVFP4 path ships it too — all keep. + + (Only migration gets this auto-keep — faithfulness wins here. On new + pages the same flags are flag-and-confirm with the maintainer: + authoring-reference §2.2 / review checklist.) + +## 2b. Playground axes: opt-out, not opt-in + +The legacy page's silence about a feature does NOT mean the axis is dropped. +Every cookbook ships the **general axes** by default — `attention` +(TP/CP/DP-Attn), `moe` (backend + EP) for MoE models, `parsers`, +`speculative`, `pdDisagg`, `hicache` — then adds model-specific axes, and +deletes ONLY axes the model genuinely cannot use (`hisparse` is DSA-only; +MegaMoE is DeepSeek-V4 Blackwell-only). Knobs meaningless for a subset of +variants/hw get `disable` + `disableReason` (per-chip constraints), not +removal — e.g. MoE backend/EP greyed out on dense variants. + +`speculative` presets must include every algorithm that actually appears on +the page — including the measured command's algorithm when it differs from +the generator default (Qwen3.5 ships both NEXTN and EAGLE) — otherwise the +verified cell's baseline can't be re-applied after a strip. + +The `parsers` axis is **add-only**: `--reasoning-parser` / +`--tool-call-parser` are never part of any Deployment cell (see §1) — the +axis adds them on top of the base command, so toggling a parser renders a +green addition, never a strikethrough. + +## 3. Verified policy mechanics + +- Green requires measured data + flag equality with the measured command (see + SKILL.md hard rule 3). Order `cells[]` so the verified flagship cell is + **first** — `cells[0]` is the page's initial selection. +- When the measured command and the generator default disagree (Qwen3.5: bench + ran `NEXTN` + `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, generator emitted `EAGLE` + + fusion flags), the verified cell mirrors the measurement; the generator + default lives on as the not-verified sibling cells. Offer BOTH as Playground + `speculative` presets and explain the split in §2 tips. +- `config.accuracyLabels` is REQUIRED whenever benchmarks carry accuracy data — + the engine ships no default eval set (#27842); without it the accuracy rows + silently don't render. `defaultAccuracy` paints every *entry-bearing* cell of + a variant — under the strict policy prefer per-entry `accuracy` on the + measured cell only. + +## 4. Per-family strategy sets (survey sketches — re-derive from the live page) + +The family table below was sketched from the 2026-06-10 survey at PAGE level. +At migration time **re-derive it from the live generator**: pages drift +(precedent: Kimi-K2.6's live page has a speculative toggle the survey notes +lack), and the per-combination rule means gated/hidden toggles — typically on +Xeon, AMD, or a single-recipe quant like NVFP4 — produce `balanced` combos the +page-level sketch doesn't show. + +**Strategy-set rule — the count follows the page's operating points** (ids +always from the DeepSeek-V4 vocabulary, never model-specific ids like +`mtp`/`no-mtp`): + +- **1 operating point** (a single recipe, no performance toggle) → a single + **`balanced`** strategy. Never invent a second recipe just to fill chips. +- **2 operating points** → **`low-latency` + `high-throughput`**. When the + legacy toggle is MTP / speculative decoding, the mapping is a + **deterministic default — apply it without asking**: MTP on → + `low-latency`, MTP off → `high-throughput`. (Why it's near-certain: + speculative decoding cuts per-token latency at low concurrency, but at + saturation the draft+verify overhead costs more than it saves — DSv4's + high-throughput recipes disable MTP for the same reason.) Other toggles + map by the same serving semantics — the two recurring **high-throughput + markers** are **dp-attention ON** (MLA-attention models) and **EP / DP+EP + ON** (MoE models): both shard work across ranks for saturated throughput + at some per-request latency cost. These directions apply to the toggle + CHOSEN as the strategy dimension (§1); a flag-only spec toggle riding + alongside a named operating-point toggle stays a Playground axis and bakes + per its legacy default — GLM-5.1's spec defaults ON, so its flags bake + into BOTH tiers there. Only if a legacy page documents the OPPOSITE slant + (e.g. "enable MTP for high throughput") stop and confirm with the + maintainer. +- **3 operating points** → the **full trio** (the ideal — e.g. GPU-budget + tiers 2/4/8). + +**Signal-driven tiers (hard rule).** A cell goes under `low-latency` / +`high-throughput` ONLY on a signal present in the legacy source: an explicit +performance toggle (MTP/speculative, dp-attention, EP, gpuCount, …), a named +recipe/strategy checkbox, option subtitles ("Low Latency" / "High +Throughput"), or prose stating the operating point. Reading such +a signal is SGLang-level serving semantics (MTP favors latency on any +vendor's silicon), so any migrator can tier any vendor's cells without +hardware-specific judgment. **No signal → `balanced`** — legacy silence is +itself information: the page offered that command as the hardware's +general-purpose operating point, and `balanced` transcribes exactly that. +Never derive a slant from your own hardware intuition ("this flag combo +feels throughput-tuned"); re-tiering on measured evidence is the hardware +owner's follow-up PR, not part of a migration. A toggle that maps to no +dimension, or a suspected undocumented slant → stop and ask the maintainer. + +The tiers apply **per (hw × variant × quant) combination**, not just per page: +a combination with fewer operating points than the page parks its cells in +the semantically honest tier. A single recipe with a signal-evidenced slant +goes to that tier (DSv4's RTX PRO 6000 → `low-latency`: workstation card, +low-batch Marlin recipe — the recipe's own SGLang-legible content is the +evidence); a general-purpose recipe with no latency/throughput slant goes to +`balanced` (Qwen3.5's Xeon → `balanced`). Never park a no-slant recipe under +`low-latency`/`high-throughput` just because the page's toggle mapping lands +there — that reads as a semantic lie ("CPU = high-throughput?"). The page's +`strategies` list is the union of tiers actually used (a mixed +[low-latency, balanced, high-throughput] page where GPUs use the two ends and +CPU uses the middle is fine); the engine greys unused chips per selection and +auto-snaps, no extra config needed. + +Deviations (e.g. how to name pure GPU-budget tiers) need maintainer sign-off. +The MDX strategy bullets describe serving semantics in the DSv4 style +(single-user chat / typical multi-user / batch jobs), with at most a one-line +model-specific note — never toggle-/migration-centric explanations. + +| Family | strategies | Notes | +|---|---|---| +| Gemma4 | `low-latency` (MTP on — the legacy toggle's own "Lower Latency" subtitle) / `high-throughput` (MTP off); mi300x hides the toggle → its single recipe → `balanced` (trio union, Qwen3.5 Xeon pattern) | variants = e2b/e4b/12b/31b/26b-a4b; checkpoint radio Standard(BF16)/QAT(q4_0) → quant ids via `modelNames`; §3.3 prose carries AMD recipes beyond the widget's mi300x — maintainer call on cells-from-prose vs tips; vision/audio invocation prose carries over (deployment matrix is text-standard); "gemma4 branch" version → speed drops, MMLU/GSM8K accuracy keeps (mind the few-shot vs run_eval harness footnote); dedicated multi-arch dev images verbatim | +| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → NEW Playground axis, engine PR FIRST** (every-feature rule §1); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); "main branch" version → speed drops, GSM8K accuracy keeps | +| GLM-4.5, GLM-4.6 | `low-latency` (TP, + MTP from the legacy checkbox) / `high-throughput` (TP+DP+EP) | | +| GLM-4.7 | `low-latency`(2 GPUs) / `balanced`(4) / `high-throughput`(8) — gpus 2/4/8 + SUPPORT matrix; confirm naming, tiers are GPU budgets | measured-best B200 TP=2 NVFP4 → the verified cell | +| GLM-4.7-Flash | `low-latency` (tp1 + MTP from the legacy checkbox) / `high-throughput` (DP) | derive from the legacy dp/mtp checkboxes | +| GLM-5, GLM-5.1 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — the dpattention radio carries the page's own "Low Latency"/"High Throughput" subtitles (naming rule, §1) | spec is flag-only, default ON, hidden on AMD → bakes into both tiers on NVIDIA + `speculative` axis; NVFP4 hides all toggles → single no-signal recipe → `balanced` (page ships the trio union) | +| Kimi-K2 | `low-latency` (tp8) / `high-throughput` (dp4+ep4) | variants = instruct/thinking; reasoning chip `hide` on instruct | +| Kimi-K2.5, K2.6 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — same named-subtitle pattern as GLM-5.1 | K2.5 spec preset carries `--speculative-draft-model-path …eagle3-mla`, chip-gated to h200/b300; K2.6's live page has a NVIDIA-only spec toggle, default OFF → `speculative` axis only, no bake (missed by the survey) | +| Qwen3.6, Qwen3-Next | `low-latency` (MTP on, the legacy speculative toggle) / `high-throughput` (MTP off); Xeon hides the toggle → its single recipe → `balanced` (page ships the trio) | same pattern as the Qwen3.5 pilot | +| Kimi-Linear, MiniMax-M2, Qwen3, Qwen3-Coder, Qwen3-Coder-Next | single `balanced` — one recipe, no performance toggle (rule above: 1 operating point → `balanced`) | renders as one chip; Qwen3-Coder-Next has NO speculative dim on the live page (quant × toolcall × mambaCache only — an earlier sketch wrongly lumped it with Qwen3.6) | +| MiniMax-M2.5, M2.7 | `low-latency`(2) / `balanced`(4) / `high-throughput`(8=tp8+ep8) — confirm naming, tiers are GPU budgets | Xeon (M2.7) is a single no-slant recipe (fixed TP=6) → `balanced` (per-combination rule, Qwen3.5 Xeon precedent) | +| Qwen3.5 (DONE — pilot) | `low-latency` (MTP on) / `high-throughput` (MTP off); Xeon's single no-slant recipe → `balanced` (the page ships the full trio) | see §5 | + +Qwen3 variant fan-out: variants = deployable checkpoints size-ordered +(`235b-instruct`, `235b-thinking`, `235b`, `30b-*`, `32b`, …); do NOT abuse +strategies for the instruct/thinking category. Trim original-hybrid chips to +the ones the legacy page actually measured. + +## 5. Worked example — the Qwen3.5 pilot (PR #27848) + +Decisions log, in the order they came up: + +1. **Strategy split over Playground toggle** because MTP couples with TP on + three H100 combos (35B/27B BF16: tp2↔tp1+mem0.88; 122B FP8: tp4↔tp2). + Canonical naming: `low-latency` = MTP on (legacy default), `high-throughput` + = MTP off. Xeon has a single operating point (the legacy widget hid the MTP + toggle there) and its recipe has no latency/throughput slant → its 12 cells + park under `balanced` (per-combination placement; parking them under + high-throughput as a toggle-mapping side effect read as a semantic lie). + Result: 186 cells = 87 low-latency + 87 high-throughput + 12 balanced; the + page ships the full trio and the engine greys unused chips per selection. +2. **Verified cell follows the measurement**: H200/397B/BF16/low-latency = + `SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm NEXTN` + (normalized spelling) + measured flag set **minus the parser flags** (the + measured run had both parsers on; cells never carry them — noted in the + benchmarks header). All other cells = the generator's parsers-OFF output + verbatim with `EAGLE`. Both spec presets exposed on the speculative axis. +3. **FP4 single quant id** with `hw|variant|quant` modelNames keys → + `nvidia/...NVFP4` (b200/b300) vs `amd/...MXFP4` (mi355x). +4. **Xeon** as `config.hardware` `vendor:"intel"`; cells carry + `--device cpu --disable-overlap-schedule`; no docker mapping. +5. **Playground axes**: the full general set per §2b — attention + (TP/CP/DP-Attn), moe (DeepEP backend + EP knob, `disable`+reason on the + dense variants), parsers, speculative (NEXTN + EAGLE — both algorithms + appear on the page), pdDisagg, hicache. Excluded as inapplicable: + hisparse (DSA-only), MegaMoE (DSv4 Blackwell-only). The legacy + `--expert-parallel-size 8` flag is normalized to `--ep 8` for the EP knob. +6. **Benchmarks**: one entry (the measured cell) only — entry-less cells render + "pending" without stubs. The legacy speed numbers were DROPPED: they were + measured on a drifting "main branch" build, which is no version anchor + (speed migrates only under an exact release tag / commit hash — hard rule + 2), so the entry carries accuracy only (GSM8K + MMMU via `accuracyLabels`, + sample counts in `notes`) and no `sglang_version`. +7. **Codegen + audit scripts** (adapt per model): a generator-port script that + emits the cells literal, and an independent audit that `git show`s the + ORIGINAL generator, stubs `useState`/`useEffect`, calls its + `generateCommand(values)` per combo via indirect eval, and token-diffs + against the new cells (expected deltas only). Read the legacy source via + `git show main:` — NOT `HEAD:` (the migration branch's HEAD has + already deleted the file, so the audit breaks after the deletion commit). + Re-run the audit after ANY later cells revision (renames included). Pilot + result: 185/185 identical + 1 intentional override. Scripts are archived + in PR #27848's description (collapsed details block). +8. **Inherited-infeasible combos kept verbatim** (e.g. 122B BF16 tp1 on + mi325x: 244 GB weights vs 256 GB VRAM with mem-fraction 0.8) — they stay + yellow and are listed in the PR body for the re-verification track. +9. **Browser-smoke probe pitfall**: multiple programmatic `.click()` calls in + one synchronous eval batch under React 18 — the DOM reads between them are + stale and look like snap-logic bugs. One click per eval, then settle. diff --git a/.claude/skills/cookbook-review-pr/SKILL.md b/.claude/skills/cookbook-review-pr/SKILL.md index 6a5bfe72b..bfcbba50f 100644 --- a/.claude/skills/cookbook-review-pr/SKILL.md +++ b/.claude/skills/cookbook-review-pr/SKILL.md @@ -48,6 +48,19 @@ than restating. (`{id,label,vram,vendor}`), **not** added to the engine catalog. - `placeholders` declares every `{{KEY}}` used in `curl` or any cell. - `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair). +- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`; + 2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per + (hw × variant × quant) combination: a single-recipe combination must park under its + semantically honest tier (clear slant → that tier, e.g. a workstation card under + `low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe + parked under low-latency/high-throughput**. Mixed unions + like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also + flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the + deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput` + (at saturation the draft+verify overhead outweighs the speedup); the reverse needs an + explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics + in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal + toggles. - `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback). - `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200` NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it). @@ -55,9 +68,13 @@ than restating. template's `model` field is a free-form input prefilled from this value; if the config omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the page's submissions get mislabeled. -- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes - (the `moe` axis's MegaMoE backend option + `megamoeQuant` block only on Blackwell MoE, - gated by `requiresHw`; `hisparse` only DSA-style; `pdDisagg.router` only with a PD topology). +- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default** + (`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`, + `pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot + use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant` + only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are + meaningless for a subset of variants/hw are `disable`d with a reason, not silently live + (e.g. MoE knobs greyed on dense variants). No empty/stub axes. - **No leftover `__TOKEN__`** — the config was stamped from the template and every placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns nothing). @@ -71,7 +88,18 @@ than restating. - NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells (the renderer injects them from `match.nodes`). - NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`. -- Flag order: `--model-path` first, then parallelism, then MoE, then tuning, `--host`/`--port` +- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a + Playground-only feature added on top of the base command (DSv4 convention); + flag any cell that bakes them in. +- Accuracy-degrading flags in cells — runtime quant below the checkpoint + (e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`) + and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV + checkpoint): **flag for explicit maintainer confirmation**. Output quality + should be exactly what the quant chip declares, so absent a recorded + sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's + default command), request the flag move to Playground/tips. +- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it — + the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port` last (the playground's insert anchors assume this). - TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom (BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params). @@ -88,6 +116,13 @@ than restating. from the `sglang serve` deploy command. - `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a guessed/placeholder value (no leftover `0.0.0`). +- **Consistent accuracy harness across entries**: every value under one `accuracyLabels` + column must be produced by the SAME harness — flag a page that, say, measures one + platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with + `run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %" + (the scores aren't comparable). Either standardize on one harness (matching + `benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding + a second contributor's measurements (e.g. an AMD/ROCm PR) into the page. ### 5. Doc ↔ config parity (the #1 finding) - Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must @@ -133,6 +168,16 @@ than restating. or `../`-relative links. `docs.sglang.io` is canonical. - No Google-Drive image links (don't render). Shell placeholders are `export VAR=`, not `${VAR}` (a bash no-op). +- **Parser ids must exist in the code registries** on the PR's target branch: every + `--reasoning-parser X` / `--tool-call-parser Y` named in prose or in + `playgroundFeatures.parsers` flags is a registered key in + `python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) / + `python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) — + prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a + factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules + (`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's + chat template to the right parser — no rule match means auto silently disables the + parser; when in doubt require explicit ids (the DSv4 page pins explicit ids). ### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md` - **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert @@ -140,17 +185,23 @@ than restating. or unknown components. ``/`` only on category `intro.mdx`, not model pages. - Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after the opening fence); a fenced block nested inside another uses four backticks outside. -- Every runnable invocation block is followed by `**Output Example:**` + a `text Output` - fenced block (real output, or `Pending update...` only with user acknowledgement). +- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable + example wrapped in an ``, its real output in a following + `` (`Pending update...` only with user + acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings. - Reasoning-parser example matches the parser's **output shape**: separate-field (`reasoning_content` + `content`) vs inline `` tags parsed out of `content`. - No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses `generation_config.json` defaults); listing them in §1 informationally is fine. ### 10. Quantization rules -- FP4 is Blackwell-only (B200/B300/GB300) — never AMD; AMD FP4 chips must be `disabled`. -- BF16 / FP8 work on NVIDIA and AMD. FP8 configs adding `--kv-cache-dtype fp8_e4m3` should - note the accuracy trade-off. +- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell + is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g. + `amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the + AMD FP4 chip must be absent/`disabled`. +- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an + accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default + home is Playground/tips). ### 11. Scope - Changes match the PR title. Flag global changes hiding behind a platform-specific title