docs: add cookbook-migrate-model skill from the Qwen3.5 pilot (#27845)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Fable 5
Claude Fable 5
parent
5633ca8599
commit
29ac249be1
@@ -6,6 +6,11 @@ disable-model-invocation: true
|
||||
|
||||
# Add a model to the SGLang Cookbook
|
||||
|
||||
> Migrating an **existing legacy-template page** (one that imports a monolithic
|
||||
> `…/autoregressive/<slug>-deployment.jsx` generator)? Use the
|
||||
> `cookbook-migrate-model` skill instead — same target format, but the legacy
|
||||
> page (not the user) is the source of truth.
|
||||
|
||||
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||
`docs_new/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||
|
||||
@@ -32,7 +32,7 @@ the full contract):
|
||||
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
|
||||
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
|
||||
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Common ids: `low-latency`, `balanced`, `high-throughput`. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
|
||||
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
|
||||
| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. |
|
||||
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
|
||||
@@ -91,6 +91,16 @@ Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
- Accuracy-degrading flags don't belong in cells by default: a cell's
|
||||
output quality should be exactly what its quantization chip declares.
|
||||
Runtime quant below the checkpoint's precision (e.g. MegaMoE **W4A4** —
|
||||
DSv4 gates it behind the Playground's `megamoeQuant` opt-in) and lossy
|
||||
KV-cache dtypes (`--kv-cache-dtype fp8_e4m3` over a higher-precision-KV
|
||||
checkpoint) default to Playground opt-ins or §2-tips material. If the
|
||||
model's recipe genuinely needs one in a cell, **flag it to the user and
|
||||
get explicit confirmation** — never ship it silently. (Migrations are the
|
||||
sanctioned exception: a flag baked into the legacy recipe's default
|
||||
command keeps verbatim — see the migrate skill.)
|
||||
|
||||
**Cells are denormalized on purpose** — common flags repeat across cells.
|
||||
This makes each cell self-contained and easy to verify. When sweeping a
|
||||
@@ -103,9 +113,14 @@ automatically.
|
||||
|
||||
## 2.3 Configure `playgroundFeatures` (optional)
|
||||
|
||||
The Playground widget is opt-in per axis. Add only the axes that make sense
|
||||
for this model. Recognised axis keys and their schemas (full reference in
|
||||
the `_playground.jsx` header):
|
||||
The Playground is **opt-out, not opt-in**: every cookbook ships the general
|
||||
axes by default — `attention` (TP/CP/DP-Attn), `moe` (backend + EP, for MoE
|
||||
models), `parsers`, `speculative`, `pdDisagg`, `hicache` — then adds
|
||||
model-specific axes (e.g. MegaMoE for DeepSeek-V4) and deletes ONLY the axes
|
||||
this model genuinely cannot use (e.g. `hisparse` on non-DSA models, `moe` on a
|
||||
pure-dense model). Knobs that don't apply to a subset of variants/hw get
|
||||
`disable` + `disableReason`, not removal. Recognised axis keys and their
|
||||
schemas (full reference in the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
|
||||
@@ -7,6 +7,10 @@ cookbook is data-only and never needs this. The current 7 built-in axes
|
||||
(MegaMoE is not its own axis — it lives inside `moe` as a backend option +
|
||||
a `megamoeQuant` sub-select.) Only add a new axis if a real cookbook needs it
|
||||
and the feature does not fit any existing axis. Touches `_playground.jsx` only.
|
||||
One routine trigger: the `cookbook-migrate-model` skill requires every legacy
|
||||
control to survive as a dimension or a Playground axis — a legacy feature no
|
||||
built-in axis covers (e.g. Nemotron3's "KV Cache DType" radio) lands here
|
||||
first, as its own engine PR ahead of the migration PR.
|
||||
|
||||
For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
|
||||
|
||||
|
||||
@@ -61,9 +61,12 @@ table is a live reference.
|
||||
- **Tool-call follow-up on thinking models**: the final assistant turn may put text in
|
||||
`reasoning_content` instead of (or with) `content` — print both so the output isn't a
|
||||
misleading `None`.
|
||||
- **Every runnable block** is immediately followed by `**Output Example:**` + a
|
||||
` ```text Output ` block with **real** server output (verbatim, not paraphrased).
|
||||
`Pending update...` is acceptable only with the user's explicit acknowledgement.
|
||||
- **§3 commands and outputs are collapsible (required)**: every runnable example
|
||||
lives in an `<Accordion title="… (Python)">` and its **real** server output
|
||||
(verbatim, not paraphrased) in an immediately following
|
||||
`<Accordion title="Example Output">` — match the DeepSeek-V4 §3 pattern. No
|
||||
inline `**Output Example:**` headings / bare blocks. `Pending update...` is
|
||||
acceptable only with the user's explicit acknowledgement.
|
||||
- **Do not hardcode sampling params** (`temperature`, `top_p`) in sample code — SGLang
|
||||
uses `generation_config.json` defaults. Listing "Recommended generation" in §1 is fine.
|
||||
- Format raw API objects (`ChatCompletionMessage(...)`) into readable Reasoning /
|
||||
|
||||
@@ -48,6 +48,12 @@ export const config = {
|
||||
{ id: "fp8", label: "FP8" },
|
||||
{ id: "fp4", label: "FP4" },
|
||||
],
|
||||
// 4th dim. The count follows the model's operating points: 1 recipe → a
|
||||
// single "balanced"; 2 → low-latency + high-throughput; 3 → the full trio
|
||||
// (the ideal). Per-combination: a single-recipe combination (e.g. a CPU
|
||||
// platform) parks under its semantically honest tier — no slant → balanced;
|
||||
// the page's list is the union and the engine greys unused chips. Never
|
||||
// invent a recipe just to fill chips.
|
||||
strategies: [
|
||||
{ id: "low-latency", label: "Low-Latency" },
|
||||
{ id: "balanced", label: "Balanced" },
|
||||
@@ -158,7 +164,10 @@ sgl-eval run gsm8k \\
|
||||
cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
|
||||
},
|
||||
|
||||
// Opt-in per axis. DELETE any axis your model doesn't expose (don't leave a stub).
|
||||
// Opt-OUT per axis: general axes (attention / moe-for-MoE / parsers /
|
||||
// speculative / pdDisagg / hicache) ship on every cookbook — DELETE only the
|
||||
// axes this model genuinely cannot use (e.g. hisparse on non-DSA models, moe
|
||||
// on pure-dense models); prefer disable+disableReason for variant/hw subsets.
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
@@ -312,6 +321,9 @@ sgl-eval run gsm8k \\
|
||||
// EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
|
||||
// your model's verified recipe, or DELETE families you don't support. `match`
|
||||
// MUST have exactly the 5 keys; env/flags are flat literals.
|
||||
// Accuracy-degrading flags (W4A4-style runtime quant, lossy --kv-cache-dtype)
|
||||
// default to Playground/tips — putting one in a cell needs explicit user
|
||||
// confirmation (authoring-reference §2.2).
|
||||
cells: [
|
||||
// ==== NVIDIA Blackwell + FP4 (single node) ====
|
||||
{
|
||||
|
||||
@@ -103,8 +103,9 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
{/* Keep only the subsections that apply. Each runnable block is followed by an
|
||||
**Output Example:** + a ```text Output block with REAL server output. */}
|
||||
{/* Keep only the subsections that apply. Commands and outputs in this section are
|
||||
COLLAPSIBLE (required — match DeepSeek-V4 §3): each runnable example lives in an
|
||||
<Accordion>, its REAL server output in a following <Accordion title="Example Output">. */}
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
@@ -114,6 +115,8 @@ Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser**
|
||||
answer → `content`). If your parser emits inline `<think>...</think>` tags inside
|
||||
`content`, parse the tags from `content` instead. */}
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
@@ -128,18 +131,23 @@ print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
TODO: paste real server output here.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
{/* TODO: tool-calling example + **Output Example:**. On thinking-mode models the
|
||||
follow-up may put text in `reasoning_content`; print both that and `content`. */}
|
||||
{/* TODO: tool-calling example in an <Accordion> + an <Accordion title="Example Output">.
|
||||
On thinking-mode models the follow-up may put text in `reasoning_content`;
|
||||
print both that and `content`. */}
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: cookbook-migrate-model
|
||||
description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs_new/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model <Model page name, e.g. GLM-5.1>.
|
||||
---
|
||||
|
||||
# Cookbook Migrate Model
|
||||
|
||||
Convert one legacy cookbook page to the config-driven format, faithfully. The
|
||||
legacy page — its generator widget and its measured benchmark blocks — is the
|
||||
**single source of truth**. You are transcribing it into the new data model,
|
||||
not improving it.
|
||||
|
||||
Reuses the `cookbook-add-model` skill's assets (read them on demand):
|
||||
- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl`
|
||||
- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract)
|
||||
- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules)
|
||||
|
||||
Migration-specific references in this skill:
|
||||
- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848).
|
||||
|
||||
The round's per-model inventory (scope, batch order, quirks, measured-data
|
||||
survey) is tracked by the migration maintainer outside the repo — expect it in
|
||||
your dispatch prompt, or ask for it.
|
||||
|
||||
## Hard rules (non-negotiable)
|
||||
|
||||
1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings
|
||||
are copied verbatim from the legacy page — even when today's defaults differ
|
||||
(e.g. `SGLANG_ENABLE_SPEC_V2=1` is now default; keep it anyway). The recipe
|
||||
that was verified is the recipe as written. Allowed normalizations are ONLY
|
||||
the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang
|
||||
serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated
|
||||
`--speculative-algo`→`--speculative-algorithm`,
|
||||
`--expert-parallel-size`→`--ep`). **Accuracy-degrading flags**
|
||||
(`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a
|
||||
deterministic rule — enforced in migration, no asking: offered as a
|
||||
legacy **selectable option** → never select it (cells mirror the
|
||||
accuracy-safe side), and the option **survives as a Playground axis** —
|
||||
an existing one where it fits, else add one via a separate prior engine
|
||||
PR (rule 4); a legacy choice never degrades to a tips mention. Baked
|
||||
into the recipe's **default/unconditional command** → keep it verbatim
|
||||
(the recipe was measured with it, and fp8 KV halves KV memory —
|
||||
stripping could OOM it). See dimension-mapping.md §2 caveats.
|
||||
2. **Never invent versions or numbers.** Benchmark numbers only from the
|
||||
legacy page's measured blocks. **Speed measurements migrate ONLY when the
|
||||
legacy page pins an exact, reproducible build** (a release tag or a commit
|
||||
hash) — drifting strings like "main branch" are no version anchor: drop the
|
||||
speed numbers AND the entry's `sglang_version`, keep accuracy (far less
|
||||
build-sensitive) and `benchmarkCommands` so ⚡Reproduce guides
|
||||
re-measurement against a pinned release; confirm ambiguous strings (e.g.
|
||||
"0.5.8+") with the maintainer. When kept, `sglang_version` is the legacy
|
||||
page's string verbatim. Docker tags only the ones the legacy page pinned
|
||||
(unmapped hw falls back to `:dev`).
|
||||
3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
|
||||
legacy page has concrete measured data for that exact 5-dim combo AND
|
||||
(b) the cell's flags equal the deployment command used for that measurement
|
||||
— modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser
|
||||
flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every
|
||||
cell (Playground-only feature; when the measured run had them on, say so in
|
||||
the benchmarks file header). When the measured command diverges from the
|
||||
generator default, **the verified cell follows the measured command**; the
|
||||
generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) —
|
||||
including combos that look memory-infeasible; keep them verbatim and list
|
||||
them in the PR body for the re-verification track.
|
||||
4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
|
||||
change in a migration PR. If the model needs an engine capability (a new
|
||||
axis, accuracy labels, …), that is a separate prior PR. The every-feature
|
||||
rule (step 2) triggers this routinely: a legacy control no built-in axis
|
||||
covers means an engine-axis PR lands first.
|
||||
5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
|
||||
`qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
|
||||
as deepseek-v4. The issue template itself needs NO edits (free-form input).
|
||||
6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` /
|
||||
`high-throughput` ONLY on a signal present in the legacy source (an
|
||||
explicit performance toggle, a named recipe, or prose stating the
|
||||
operating point); **no signal → `balanced`**. Never derive a slant from
|
||||
your own hardware intuition — re-tiering on measured evidence is the
|
||||
hardware owner's follow-up PR, not part of a migration
|
||||
(dimension-mapping.md §4).
|
||||
|
||||
## Workflow (one model = one PR)
|
||||
|
||||
### 1. Inventory the legacy assets
|
||||
- Read the legacy generator (`docs_new/src/snippets/autoregressive/<slug>-deployment.jsx`)
|
||||
end-to-end: every option dimension (radio vs checkbox vs dynamic), every
|
||||
gate/SUPPORT matrix, the full emitted command per reachable combo (env
|
||||
prefixes, `# Error` pseudo-commands included).
|
||||
- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not
|
||||
upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
|
||||
Output Examples verbatim); §5 benchmark blocks → transcribe each measured
|
||||
block: deploy command used, bench command (dataset/isl/osl/num-prompts/
|
||||
concurrency), Mean TTFT/TPOT, output tok/s, hardware, version string.
|
||||
- Inbound-anchor sweep: `grep -rn "<PageName>" docs_new/ --include='*.mdx'` —
|
||||
find links/`#fragments` into this page (`mint broken-links` does NOT check
|
||||
fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
|
||||
- Check the maintainer-provided inventory notes for this model's known quirks —
|
||||
but treat them (and the §4 family table) as a **survey snapshot**: re-verify
|
||||
every dimension against the live legacy files before mapping. Pages keep
|
||||
receiving updates (precedent: Kimi-K2.6's live generator has a speculative
|
||||
toggle the 2026-06-10 survey notes lack).
|
||||
|
||||
### 2. Design the 5-dim mapping
|
||||
Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key
|
||||
decision — which legacy toggle becomes the `strategies` dimension: a toggle
|
||||
that **changes other parts of the command** (TP, mem) must (the Playground
|
||||
can't do coupled changes), and so does a toggle the legacy page itself labels
|
||||
with operating-point words — e.g. a `dpattention` radio whose options are
|
||||
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
|
||||
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
|
||||
flag add). Any other toggle that only adds/removes its own flags becomes a
|
||||
Playground axis with the flags baked into cells when the legacy default was
|
||||
ON — EXCEPT parsers:
|
||||
`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are
|
||||
Playground-only (DSv4 convention). **Every legacy control survives as an
|
||||
interactive control** — a dimension or a Playground axis, never a tips-only
|
||||
mention. A feature none of the built-in axes covers (Nemotron3's "KV Cache
|
||||
DType" radio is the precedent) still lands in the Playground: add the axis
|
||||
via a separate PRIOR engine PR (engine-axis.md), keeping the migration PR
|
||||
itself data-only (hard rule 4). The strategy count follows the page's
|
||||
operating points: **one recipe → a single `balanced`; two → `low-latency` +
|
||||
`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
|
||||
(hw × variant × quant) combination — a single-recipe combination on a
|
||||
multi-strategy page parks under its semantically honest tier (no
|
||||
latency/throughput slant → `balanced`; the page's list is the union). When the
|
||||
legacy toggle is MTP / speculative decoding, the direction is a deterministic
|
||||
default — apply without asking: **MTP on → `low-latency`, MTP off →
|
||||
`high-throughput`** (reversed only with maintainer confirmation). Tier
|
||||
placement is signal-driven (hard rule 6). Never invent a recipe just to fill
|
||||
strategy chips (see dimension-mapping.md §4). Record the outcome as a
|
||||
**strategy mapping table** for the PR body — one row per group of
|
||||
combinations sharing the same legacy signal (e.g. "all GPU combos: MTP
|
||||
toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a
|
||||
one-line rationale each; don't enumerate 60 identical rows. The table is what
|
||||
hardware owners sign off on at review.
|
||||
|
||||
### 3. Generate the config (codegen, then audit)
|
||||
- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node
|
||||
script that enumerates combos and emits the `cells:[...]` literal (output
|
||||
must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the
|
||||
verified-cell override in the script. See the pilot scripts embedded in the
|
||||
worked example of dimension-mapping.md §5.
|
||||
- **Independent equivalence audit (required):** extract the ORIGINAL generator
|
||||
from git (`git show main:<path>` — NOT `HEAD:`, the migration branch deletes
|
||||
the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for
|
||||
every combo, and diff token-by-token against the new cells. Expected deltas
|
||||
only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the
|
||||
engine-injected multi-node trio, the §2 alias rewrites (the entrypoint
|
||||
rewrite doesn't appear in cell tokens — cells hold flags only; the audit
|
||||
script normalizes it on the legacy side), and the intentional verified-cell
|
||||
override. Paste the PASS count + the audit script in the PR body
|
||||
(collapsed `<details>`).
|
||||
- Hand-author the non-cells fields per authoring-reference.md. Structural
|
||||
self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/
|
||||
--dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every
|
||||
`supportedHardware` id has ≥1 cell.
|
||||
|
||||
### 4. Benchmarks file
|
||||
One entry per measured block only (cells without entries already render
|
||||
"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
|
||||
output tok/s ÷ (tp × nnodes); TTFT/TPOT take the Mean rows; put the workload's
|
||||
`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
|
||||
the benchmarks carry accuracy data** — the engine ships no default eval set
|
||||
(#27842), so missing labels means the accuracy rows silently don't render;
|
||||
extra context (sample counts, suites that don't fit) goes in the entry's
|
||||
`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop
|
||||
entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users.
|
||||
|
||||
### 5. Rewrite the MDX
|
||||
From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh
|
||||
SEO `description` (top-level — delete any legacy `metatags.description`), **no
|
||||
`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion
|
||||
carries the legacy install content + pinned images. Keep the template's
|
||||
DSv4-style strategy bullets — serving semantics first (single-user chat /
|
||||
typical multi-user / batch throughput), trimmed to the strategies the page
|
||||
ships, plus at most a one-line note on what each strategy changes on this
|
||||
model; do NOT rewrite them as toggle-/migration-centric explanations.
|
||||
Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands →
|
||||
⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity —
|
||||
fold their unique flags into §2 tips). Invocation examples + real outputs carry
|
||||
over verbatim, but **wrapped in Accordions** — §3 commands and outputs are
|
||||
collapsible (required, DeepSeek-V4 pattern): code in an
|
||||
`<Accordion title="… (Python)">`, output in a following
|
||||
`<Accordion title="Example Output">`; legacy pages kept them inline.
|
||||
|
||||
### 6. Delete the legacy generator
|
||||
Remove `docs_new/src/snippets/autoregressive/<slug>-deployment.jsx` and its
|
||||
import. `grep -rn "<slug>-deployment" docs_new/` must return nothing (config
|
||||
provenance comments must not name the deleted path). Site wiring needs **no
|
||||
changes**: docs.json path/title unchanged, vendor card + logo already exist.
|
||||
|
||||
### 7. Validate
|
||||
- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens).
|
||||
- `cd docs_new && mint validate && mint broken-links` (pre-existing breaks on
|
||||
main are not yours — say so in the PR).
|
||||
- `mint dev` browser smoke: initial selection = the verified cell (first in
|
||||
`cells[]`) with green badge; multi-node cells show the injected trio +
|
||||
header; AMD cells show env prefixes; Docker mode wraps with the pinned image
|
||||
and passes cell env as `--env`; condition-hidden combos grey out; benchmark
|
||||
card values; NO parser flags in any Deploy command; Playground parser
|
||||
toggles ADD the parser flags (green additions) while spec toggles strike
|
||||
the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most
|
||||
ONE programmatic click per evaluation and wait for React to settle —
|
||||
multiple clicks in one synchronous script batch and read stale DOM.
|
||||
- Token-level audit from step 3 passes.
|
||||
|
||||
### 8. PR + review
|
||||
One PR per model. PR body: migration framing, verified policy applied, the
|
||||
strategy mapping table (step 2), the audit PASS count + script, any
|
||||
inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr <N>` and fix findings.
|
||||
FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches —
|
||||
a fork-headed PR is perfectly fine but renders no preview; a maintainer can
|
||||
re-push the branch in-repo if a preview is wanted for review.
|
||||
|
||||
### 9. Keep this skill current
|
||||
Any new convention, engine behavior, or pitfall you discover while migrating
|
||||
(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed
|
||||
back into this skill — same PR if it's skill-file-only, or an immediate
|
||||
follow-up commit on the skill's branch/PR. The next agent runs on what's
|
||||
written here, not on your session's context.
|
||||
@@ -0,0 +1,240 @@
|
||||
# Legacy → config-driven dimension mapping
|
||||
|
||||
Loaded on demand by the `cookbook-migrate-model` skill. How to translate a
|
||||
legacy generator's option space into the 5-dim matrix + Playground axes.
|
||||
Field schemas live in `../../cookbook-add-model/references/authoring-reference.md`;
|
||||
this file is about the *mapping decisions*.
|
||||
|
||||
## 1. Legacy control → new home
|
||||
|
||||
| Legacy control | New home | Rule |
|
||||
|---|---|---|
|
||||
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **No fitting built-in axis ≠ drop the feature** — EVERY legacy control survives as an interactive control (a dimension or a Playground axis), never a tips-only mention. When none of the built-in axes fits (Nemotron3's "KV Cache DType" radio is the precedent), add the axis via the engine-axis flow: a separate PRIOR engine PR, then the migration config declares it (hard rule 4). |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending); Qwen3.6 / Qwen3-Coder-Next carry the same knob, so their migrations need the axis in place first. |
|
||||
|
||||
## 2. Command rewrite table (the ONLY allowed normalizations)
|
||||
|
||||
| Legacy | New |
|
||||
|---|---|
|
||||
| `python(3) -m sglang.launch_server` | (engine emits `sglang serve`; cells hold flags only) |
|
||||
| `--model X` / `--model-path X` | `--model-path {{MODEL_NAME}}` + `modelNames` key |
|
||||
| `--tp-size N` | `--tp N` |
|
||||
| `--speculative-algo X` (abbreviated) | `--speculative-algorithm X` — the Playground spec axis strips/derives by the full first token only; an abbreviated alias would survive toggles and double up |
|
||||
| `--expert-parallel-size N` | `--ep N` — the Playground EP knob recognizes/strips only `--ep`; the long form would survive toggles and double up |
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
|
||||
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
|
||||
|
||||
Caveats discovered in the pilot:
|
||||
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
|
||||
`--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob
|
||||
can recognize/strip it.
|
||||
- `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class);
|
||||
standard-IB H100 multi-node needs none.
|
||||
- `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped
|
||||
(`:dev` fallback) with a "install from source" tip.
|
||||
- **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style
|
||||
runtime quant) — deterministic rule, enforced in migration without
|
||||
asking:
|
||||
- offered as a legacy **selectable option/toggle** → never select it;
|
||||
cells mirror the accuracy-safe side (even if the legacy default was the
|
||||
lossy side). The option itself **must survive as a Playground axis** — the user's choice may not degrade to a
|
||||
tips mention. Use the existing axis when one fits (DSv4 gates W4A4
|
||||
behind `megamoeQuant`); when the playground lacks one — e.g.
|
||||
Nemotron3-Ultra's "KV Cache DType" radio (None default / fp8_e4m3 /
|
||||
bf16) has no kv-cache axis today — **add the axis** via the engine-axis
|
||||
flow: a separate PRIOR engine PR (hard rule 4, like accuracyLabels
|
||||
#27842), then the migration config declares it;
|
||||
- baked into the recipe's **unconditional/default command** → keep it
|
||||
verbatim. The legacy measurements ran with it, and fp8 KV halves KV
|
||||
memory — stripping could OOM the recipe. Expect this pattern: legacy
|
||||
AMD recipes routinely append `--kv-cache-dtype fp8_e4m3` ("for memory
|
||||
efficiency"), and GLM-5's NVFP4 path ships it too — all keep.
|
||||
|
||||
(Only migration gets this auto-keep — faithfulness wins here. On new
|
||||
pages the same flags are flag-and-confirm with the maintainer:
|
||||
authoring-reference §2.2 / review checklist.)
|
||||
|
||||
## 2b. Playground axes: opt-out, not opt-in
|
||||
|
||||
The legacy page's silence about a feature does NOT mean the axis is dropped.
|
||||
Every cookbook ships the **general axes** by default — `attention`
|
||||
(TP/CP/DP-Attn), `moe` (backend + EP) for MoE models, `parsers`,
|
||||
`speculative`, `pdDisagg`, `hicache` — then adds model-specific axes, and
|
||||
deletes ONLY axes the model genuinely cannot use (`hisparse` is DSA-only;
|
||||
MegaMoE is DeepSeek-V4 Blackwell-only). Knobs meaningless for a subset of
|
||||
variants/hw get `disable` + `disableReason` (per-chip constraints), not
|
||||
removal — e.g. MoE backend/EP greyed out on dense variants.
|
||||
|
||||
`speculative` presets must include every algorithm that actually appears on
|
||||
the page — including the measured command's algorithm when it differs from
|
||||
the generator default (Qwen3.5 ships both NEXTN and EAGLE) — otherwise the
|
||||
verified cell's baseline can't be re-applied after a strip.
|
||||
|
||||
The `parsers` axis is **add-only**: `--reasoning-parser` /
|
||||
`--tool-call-parser` are never part of any Deployment cell (see §1) — the
|
||||
axis adds them on top of the base command, so toggling a parser renders a
|
||||
green addition, never a strikethrough.
|
||||
|
||||
## 3. Verified policy mechanics
|
||||
|
||||
- Green requires measured data + flag equality with the measured command (see
|
||||
SKILL.md hard rule 3). Order `cells[]` so the verified flagship cell is
|
||||
**first** — `cells[0]` is the page's initial selection.
|
||||
- When the measured command and the generator default disagree (Qwen3.5: bench
|
||||
ran `NEXTN` + `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, generator emitted `EAGLE` +
|
||||
fusion flags), the verified cell mirrors the measurement; the generator
|
||||
default lives on as the not-verified sibling cells. Offer BOTH as Playground
|
||||
`speculative` presets and explain the split in §2 tips.
|
||||
- `config.accuracyLabels` is REQUIRED whenever benchmarks carry accuracy data —
|
||||
the engine ships no default eval set (#27842); without it the accuracy rows
|
||||
silently don't render. `defaultAccuracy` paints every *entry-bearing* cell of
|
||||
a variant — under the strict policy prefer per-entry `accuracy` on the
|
||||
measured cell only.
|
||||
|
||||
## 4. Per-family strategy sets (survey sketches — re-derive from the live page)
|
||||
|
||||
The family table below was sketched from the 2026-06-10 survey at PAGE level.
|
||||
At migration time **re-derive it from the live generator**: pages drift
|
||||
(precedent: Kimi-K2.6's live page has a speculative toggle the survey notes
|
||||
lack), and the per-combination rule means gated/hidden toggles — typically on
|
||||
Xeon, AMD, or a single-recipe quant like NVFP4 — produce `balanced` combos the
|
||||
page-level sketch doesn't show.
|
||||
|
||||
**Strategy-set rule — the count follows the page's operating points** (ids
|
||||
always from the DeepSeek-V4 vocabulary, never model-specific ids like
|
||||
`mtp`/`no-mtp`):
|
||||
|
||||
- **1 operating point** (a single recipe, no performance toggle) → a single
|
||||
**`balanced`** strategy. Never invent a second recipe just to fill chips.
|
||||
- **2 operating points** → **`low-latency` + `high-throughput`**. When the
|
||||
legacy toggle is MTP / speculative decoding, the mapping is a
|
||||
**deterministic default — apply it without asking**: MTP on →
|
||||
`low-latency`, MTP off → `high-throughput`. (Why it's near-certain:
|
||||
speculative decoding cuts per-token latency at low concurrency, but at
|
||||
saturation the draft+verify overhead costs more than it saves — DSv4's
|
||||
high-throughput recipes disable MTP for the same reason.) Other toggles
|
||||
map by the same serving semantics — the two recurring **high-throughput
|
||||
markers** are **dp-attention ON** (MLA-attention models) and **EP / DP+EP
|
||||
ON** (MoE models): both shard work across ranks for saturated throughput
|
||||
at some per-request latency cost. These directions apply to the toggle
|
||||
CHOSEN as the strategy dimension (§1); a flag-only spec toggle riding
|
||||
alongside a named operating-point toggle stays a Playground axis and bakes
|
||||
per its legacy default — GLM-5.1's spec defaults ON, so its flags bake
|
||||
into BOTH tiers there. Only if a legacy page documents the OPPOSITE slant
|
||||
(e.g. "enable MTP for high throughput") stop and confirm with the
|
||||
maintainer.
|
||||
- **3 operating points** → the **full trio** (the ideal — e.g. GPU-budget
|
||||
tiers 2/4/8).
|
||||
|
||||
**Signal-driven tiers (hard rule).** A cell goes under `low-latency` /
|
||||
`high-throughput` ONLY on a signal present in the legacy source: an explicit
|
||||
performance toggle (MTP/speculative, dp-attention, EP, gpuCount, …), a named
|
||||
recipe/strategy checkbox, option subtitles ("Low Latency" / "High
|
||||
Throughput"), or prose stating the operating point. Reading such
|
||||
a signal is SGLang-level serving semantics (MTP favors latency on any
|
||||
vendor's silicon), so any migrator can tier any vendor's cells without
|
||||
hardware-specific judgment. **No signal → `balanced`** — legacy silence is
|
||||
itself information: the page offered that command as the hardware's
|
||||
general-purpose operating point, and `balanced` transcribes exactly that.
|
||||
Never derive a slant from your own hardware intuition ("this flag combo
|
||||
feels throughput-tuned"); re-tiering on measured evidence is the hardware
|
||||
owner's follow-up PR, not part of a migration. A toggle that maps to no
|
||||
dimension, or a suspected undocumented slant → stop and ask the maintainer.
|
||||
|
||||
The tiers apply **per (hw × variant × quant) combination**, not just per page:
|
||||
a combination with fewer operating points than the page parks its cells in
|
||||
the semantically honest tier. A single recipe with a signal-evidenced slant
|
||||
goes to that tier (DSv4's RTX PRO 6000 → `low-latency`: workstation card,
|
||||
low-batch Marlin recipe — the recipe's own SGLang-legible content is the
|
||||
evidence); a general-purpose recipe with no latency/throughput slant goes to
|
||||
`balanced` (Qwen3.5's Xeon → `balanced`). Never park a no-slant recipe under
|
||||
`low-latency`/`high-throughput` just because the page's toggle mapping lands
|
||||
there — that reads as a semantic lie ("CPU = high-throughput?"). The page's
|
||||
`strategies` list is the union of tiers actually used (a mixed
|
||||
[low-latency, balanced, high-throughput] page where GPUs use the two ends and
|
||||
CPU uses the middle is fine); the engine greys unused chips per selection and
|
||||
auto-snaps, no extra config needed.
|
||||
|
||||
Deviations (e.g. how to name pure GPU-budget tiers) need maintainer sign-off.
|
||||
The MDX strategy bullets describe serving semantics in the DSv4 style
|
||||
(single-user chat / typical multi-user / batch jobs), with at most a one-line
|
||||
model-specific note — never toggle-/migration-centric explanations.
|
||||
|
||||
| Family | strategies | Notes |
|
||||
|---|---|---|
|
||||
| Gemma4 | `low-latency` (MTP on — the legacy toggle's own "Lower Latency" subtitle) / `high-throughput` (MTP off); mi300x hides the toggle → its single recipe → `balanced` (trio union, Qwen3.5 Xeon pattern) | variants = e2b/e4b/12b/31b/26b-a4b; checkpoint radio Standard(BF16)/QAT(q4_0) → quant ids via `modelNames`; §3.3 prose carries AMD recipes beyond the widget's mi300x — maintainer call on cells-from-prose vs tips; vision/audio invocation prose carries over (deployment matrix is text-standard); "gemma4 branch" version → speed drops, MMLU/GSM8K accuracy keeps (mind the few-shot vs run_eval harness footnote); dedicated multi-arch dev images verbatim |
|
||||
| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → NEW Playground axis, engine PR FIRST** (every-feature rule §1); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); "main branch" version → speed drops, GSM8K accuracy keeps |
|
||||
| GLM-4.5, GLM-4.6 | `low-latency` (TP, + MTP from the legacy checkbox) / `high-throughput` (TP+DP+EP) | |
|
||||
| GLM-4.7 | `low-latency`(2 GPUs) / `balanced`(4) / `high-throughput`(8) — gpus 2/4/8 + SUPPORT matrix; confirm naming, tiers are GPU budgets | measured-best B200 TP=2 NVFP4 → the verified cell |
|
||||
| GLM-4.7-Flash | `low-latency` (tp1 + MTP from the legacy checkbox) / `high-throughput` (DP) | derive from the legacy dp/mtp checkboxes |
|
||||
| GLM-5, GLM-5.1 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — the dpattention radio carries the page's own "Low Latency"/"High Throughput" subtitles (naming rule, §1) | spec is flag-only, default ON, hidden on AMD → bakes into both tiers on NVIDIA + `speculative` axis; NVFP4 hides all toggles → single no-signal recipe → `balanced` (page ships the trio union) |
|
||||
| Kimi-K2 | `low-latency` (tp8) / `high-throughput` (dp4+ep4) | variants = instruct/thinking; reasoning chip `hide` on instruct |
|
||||
| Kimi-K2.5, K2.6 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — same named-subtitle pattern as GLM-5.1 | K2.5 spec preset carries `--speculative-draft-model-path …eagle3-mla`, chip-gated to h200/b300; K2.6's live page has a NVIDIA-only spec toggle, default OFF → `speculative` axis only, no bake (missed by the survey) |
|
||||
| Qwen3.6, Qwen3-Next | `low-latency` (MTP on, the legacy speculative toggle) / `high-throughput` (MTP off); Xeon hides the toggle → its single recipe → `balanced` (page ships the trio) | same pattern as the Qwen3.5 pilot |
|
||||
| Kimi-Linear, MiniMax-M2, Qwen3, Qwen3-Coder, Qwen3-Coder-Next | single `balanced` — one recipe, no performance toggle (rule above: 1 operating point → `balanced`) | renders as one chip; Qwen3-Coder-Next has NO speculative dim on the live page (quant × toolcall × mambaCache only — an earlier sketch wrongly lumped it with Qwen3.6) |
|
||||
| MiniMax-M2.5, M2.7 | `low-latency`(2) / `balanced`(4) / `high-throughput`(8=tp8+ep8) — confirm naming, tiers are GPU budgets | Xeon (M2.7) is a single no-slant recipe (fixed TP=6) → `balanced` (per-combination rule, Qwen3.5 Xeon precedent) |
|
||||
| Qwen3.5 (DONE — pilot) | `low-latency` (MTP on) / `high-throughput` (MTP off); Xeon's single no-slant recipe → `balanced` (the page ships the full trio) | see §5 |
|
||||
|
||||
Qwen3 variant fan-out: variants = deployable checkpoints size-ordered
|
||||
(`235b-instruct`, `235b-thinking`, `235b`, `30b-*`, `32b`, …); do NOT abuse
|
||||
strategies for the instruct/thinking category. Trim original-hybrid chips to
|
||||
the ones the legacy page actually measured.
|
||||
|
||||
## 5. Worked example — the Qwen3.5 pilot (PR #27848)
|
||||
|
||||
Decisions log, in the order they came up:
|
||||
|
||||
1. **Strategy split over Playground toggle** because MTP couples with TP on
|
||||
three H100 combos (35B/27B BF16: tp2↔tp1+mem0.88; 122B FP8: tp4↔tp2).
|
||||
Canonical naming: `low-latency` = MTP on (legacy default), `high-throughput`
|
||||
= MTP off. Xeon has a single operating point (the legacy widget hid the MTP
|
||||
toggle there) and its recipe has no latency/throughput slant → its 12 cells
|
||||
park under `balanced` (per-combination placement; parking them under
|
||||
high-throughput as a toggle-mapping side effect read as a semantic lie).
|
||||
Result: 186 cells = 87 low-latency + 87 high-throughput + 12 balanced; the
|
||||
page ships the full trio and the engine greys unused chips per selection.
|
||||
2. **Verified cell follows the measurement**: H200/397B/BF16/low-latency =
|
||||
`SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm NEXTN`
|
||||
(normalized spelling) + measured flag set **minus the parser flags** (the
|
||||
measured run had both parsers on; cells never carry them — noted in the
|
||||
benchmarks header). All other cells = the generator's parsers-OFF output
|
||||
verbatim with `EAGLE`. Both spec presets exposed on the speculative axis.
|
||||
3. **FP4 single quant id** with `hw|variant|quant` modelNames keys →
|
||||
`nvidia/...NVFP4` (b200/b300) vs `amd/...MXFP4` (mi355x).
|
||||
4. **Xeon** as `config.hardware` `vendor:"intel"`; cells carry
|
||||
`--device cpu --disable-overlap-schedule`; no docker mapping.
|
||||
5. **Playground axes**: the full general set per §2b — attention
|
||||
(TP/CP/DP-Attn), moe (DeepEP backend + EP knob, `disable`+reason on the
|
||||
dense variants), parsers, speculative (NEXTN + EAGLE — both algorithms
|
||||
appear on the page), pdDisagg, hicache. Excluded as inapplicable:
|
||||
hisparse (DSA-only), MegaMoE (DSv4 Blackwell-only). The legacy
|
||||
`--expert-parallel-size 8` flag is normalized to `--ep 8` for the EP knob.
|
||||
6. **Benchmarks**: one entry (the measured cell) only — entry-less cells render
|
||||
"pending" without stubs. The legacy speed numbers were DROPPED: they were
|
||||
measured on a drifting "main branch" build, which is no version anchor
|
||||
(speed migrates only under an exact release tag / commit hash — hard rule
|
||||
2), so the entry carries accuracy only (GSM8K + MMMU via `accuracyLabels`,
|
||||
sample counts in `notes`) and no `sglang_version`.
|
||||
7. **Codegen + audit scripts** (adapt per model): a generator-port script that
|
||||
emits the cells literal, and an independent audit that `git show`s the
|
||||
ORIGINAL generator, stubs `useState`/`useEffect`, calls its
|
||||
`generateCommand(values)` per combo via indirect eval, and token-diffs
|
||||
against the new cells (expected deltas only). Read the legacy source via
|
||||
`git show main:<path>` — NOT `HEAD:` (the migration branch's HEAD has
|
||||
already deleted the file, so the audit breaks after the deletion commit).
|
||||
Re-run the audit after ANY later cells revision (renames included). Pilot
|
||||
result: 185/185 identical + 1 intentional override. Scripts are archived
|
||||
in PR #27848's description (collapsed details block).
|
||||
8. **Inherited-infeasible combos kept verbatim** (e.g. 122B BF16 tp1 on
|
||||
mi325x: 244 GB weights vs 256 GB VRAM with mem-fraction 0.8) — they stay
|
||||
yellow and are listed in the PR body for the re-verification track.
|
||||
9. **Browser-smoke probe pitfall**: multiple programmatic `.click()` calls in
|
||||
one synchronous eval batch under React 18 — the DOM reads between them are
|
||||
stale and look like snap-logic bugs. One click per eval, then settle.
|
||||
@@ -48,6 +48,19 @@ than restating.
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
|
||||
2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per
|
||||
(hw × variant × quant) combination: a single-recipe combination must park under its
|
||||
semantically honest tier (clear slant → that tier, e.g. a workstation card under
|
||||
`low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe
|
||||
parked under low-latency/high-throughput**. Mixed unions
|
||||
like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also
|
||||
flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the
|
||||
deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput`
|
||||
(at saturation the draft+verify overhead outweighs the speedup); the reverse needs an
|
||||
explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
|
||||
in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
|
||||
toggles.
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
@@ -55,9 +68,13 @@ than restating.
|
||||
template's `model` field is a free-form input prefilled from this value; if the config
|
||||
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||
page's submissions get mislabeled.
|
||||
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
|
||||
(the `moe` axis's MegaMoE backend option + `megamoeQuant` block only on Blackwell MoE,
|
||||
gated by `requiresHw`; `hisparse` only DSA-style; `pdDisagg.router` only with a PD topology).
|
||||
- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default**
|
||||
(`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`,
|
||||
`pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot
|
||||
use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant`
|
||||
only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are
|
||||
meaningless for a subset of variants/hw are `disable`d with a reason, not silently live
|
||||
(e.g. MoE knobs greyed on dense variants). No empty/stub axes.
|
||||
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||
nothing).
|
||||
@@ -71,7 +88,18 @@ than restating.
|
||||
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||
(the renderer injects them from `match.nodes`).
|
||||
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||
- Flag order: `--model-path` first, then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a
|
||||
Playground-only feature added on top of the base command (DSv4 convention);
|
||||
flag any cell that bakes them in.
|
||||
- Accuracy-degrading flags in cells — runtime quant below the checkpoint
|
||||
(e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`)
|
||||
and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV
|
||||
checkpoint): **flag for explicit maintainer confirmation**. Output quality
|
||||
should be exactly what the quant chip declares, so absent a recorded
|
||||
sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's
|
||||
default command), request the flag move to Playground/tips.
|
||||
- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it —
|
||||
the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
last (the playground's insert anchors assume this).
|
||||
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||
@@ -88,6 +116,13 @@ than restating.
|
||||
from the `sglang serve` deploy command.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
|
||||
column must be produced by the SAME harness — flag a page that, say, measures one
|
||||
platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
|
||||
`run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
|
||||
(the scores aren't comparable). Either standardize on one harness (matching
|
||||
`benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
|
||||
a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
|
||||
|
||||
### 5. Doc ↔ config parity (the #1 finding)
|
||||
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||
@@ -133,6 +168,16 @@ than restating.
|
||||
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||
not `${VAR}` (a bash no-op).
|
||||
- **Parser ids must exist in the code registries** on the PR's target branch: every
|
||||
`--reasoning-parser X` / `--tool-call-parser Y` named in prose or in
|
||||
`playgroundFeatures.parsers` flags is a registered key in
|
||||
`python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) /
|
||||
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||
(`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
|
||||
chat template to the right parser — no rule match means auto silently disables the
|
||||
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||
|
||||
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||
@@ -140,17 +185,23 @@ than restating.
|
||||
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||
- Every runnable invocation block is followed by `**Output Example:**` + a `text Output`
|
||||
fenced block (real output, or `Pending update...` only with user acknowledgement).
|
||||
- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable
|
||||
example wrapped in an `<Accordion>`, its real output in a following
|
||||
`<Accordion title="Example Output">` (`Pending update...` only with user
|
||||
acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings.
|
||||
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||
|
||||
### 10. Quantization rules
|
||||
- FP4 is Blackwell-only (B200/B300/GB300) — never AMD; AMD FP4 chips must be `disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. FP8 configs adding `--kv-cache-dtype fp8_e4m3` should
|
||||
note the accuracy trade-off.
|
||||
- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell
|
||||
is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g.
|
||||
`amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the
|
||||
AMD FP4 chip must be absent/`disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an
|
||||
accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default
|
||||
home is Playground/tips).
|
||||
|
||||
### 11. Scope
|
||||
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||
|
||||
Reference in New Issue
Block a user