Cookbook renovation (#26885)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
6365d6faee
commit
d1777d1f6d
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: cookbook-add-model
|
||||
description: Add a new model to the SGLang Cookbook (docs_new/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Add a model to the SGLang Cookbook
|
||||
|
||||
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||
`docs_new/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||
a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
|
||||
that imports them. No engine edits.
|
||||
|
||||
**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
|
||||
template is decoupled and covers all hardware + all axes):
|
||||
- `templates/config.jsx.tmpl` → `docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx`
|
||||
- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
|
||||
- `templates/page.mdx.tmpl` → `docs_new/cookbook/<category>/<Vendor>/<ModelName>.mdx`
|
||||
|
||||
The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
|
||||
lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
|
||||
*instance* you can consult, but is not the template.
|
||||
|
||||
**Deep references (read on demand, don't inline):**
|
||||
- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
|
||||
- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
|
||||
- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
|
||||
- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ cookbook/<category>/<Vendor>/<Model>.mdx │
|
||||
│ import { Deployment } from "/src/snippets/_deployment.jsx"; │
|
||||
│ import { Playground } from "/src/snippets/_playground.jsx"; │
|
||||
│ import { config } from "/src/snippets/configs/.../X.jsx"; │
|
||||
│ <Deployment config={config} /> <Playground config={config} />│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (config passed as React prop)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ src/snippets/configs/<vendor>/<model>.jsx │
|
||||
│ export const config = { │
|
||||
│ supportedHardware, variants, quantizations, strategies, ... │
|
||||
│ cells: [ { match:{hw,variant,quant,strategy,nodes}, │
|
||||
│ env:[...], flags:[...] }, ... ], // 5-dim matrix │
|
||||
│ playgroundFeatures: { attention, moe, parsers, ... }, │
|
||||
│ }; │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (consumed by BOTH engines — no model code in engines)
|
||||
▼
|
||||
┌──────────────────────────────────┬──────────────────────────────┐
|
||||
│ _deployment.jsx │ _playground.jsx │
|
||||
│ Renders the verified matrix; │ Renders override chips + │
|
||||
│ one cell → its env/flags. │ diff against the cell. │
|
||||
└──────────────────────────────────┴──────────────────────────────┘
|
||||
```
|
||||
|
||||
The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
|
||||
playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
|
||||
every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
|
||||
shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
|
||||
|
||||
> The template is **autoregressive**. Diffusion / omni pages follow their own category
|
||||
> structure — don't force the config-driven template on them; still obey the Mintlify /
|
||||
> NEW-tag / docs.json / category-card / validation rules below.
|
||||
|
||||
---
|
||||
|
||||
**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
|
||||
everything upfront.** The real work is the verified `cells[]` recipes + measured
|
||||
benchmarks (Phases 2 + 4); everything else is filling the template.
|
||||
|
||||
## Phase 1 — Collect inputs
|
||||
|
||||
1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
|
||||
param count, architecture, context length, **license**. (Fetching guards factual bugs
|
||||
like an off-by-a-few-B param count.) If the model isn't public, ask the user.
|
||||
2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
|
||||
Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
|
||||
Default to BF16 when a full-precision repo exists.
|
||||
3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
|
||||
for each. List only tested hw (unlisted greys out).
|
||||
4. **Verified launch recipes** — the full `sglang serve` flags per
|
||||
(hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
|
||||
`python -m sglang.launch_server` to `sglang serve` form.
|
||||
5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
|
||||
(a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
|
||||
tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
|
||||
default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
|
||||
6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
|
||||
|
||||
**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
|
||||
this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
|
||||
(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
|
||||
|
||||
| Platform | Vendor | VRAM | Docker image |
|
||||
|---|---|---|---|
|
||||
| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
|
||||
| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
|
||||
| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
|
||||
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
|
||||
- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
|
||||
default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
|
||||
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
|
||||
hosts are typically **4 GPUs** (TP=4 ceiling).
|
||||
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||
- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
|
||||
`--ep` unless benchmarked. (The template's AMD example cell shows these.)
|
||||
|
||||
## Phase 2 — Instantiate the template
|
||||
|
||||
1. **Copy** the three template files to their target paths (above). Note the two
|
||||
vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
|
||||
2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
|
||||
`__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
|
||||
none remain: `grep -rn '__[A-Z_]*__' <new files>`.
|
||||
3. **Prune** to what the model supports (delete, don't stub) — using
|
||||
[references/authoring-reference.md](references/authoring-reference.md):
|
||||
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||
- `playgroundFeatures` axes: delete `megamoe` (non-Blackwell-MoE), `hisparse`
|
||||
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||
`modelNames`/`defaultAccuracy` keys).
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
- **`docs_new/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
|
||||
the **top** of that vendor's `pages` (root-relative, no `.mdx`:
|
||||
`cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
|
||||
local ordering.
|
||||
- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
|
||||
vendor dir for existing NEW and strip it from siblings; verify ≤1:
|
||||
`grep -rn 'tag: NEW' docs_new/cookbook/<category>/<Vendor>/` → at most one result. (Scan
|
||||
files; don't assume the first `docs.json` entry holds NEW.)
|
||||
- **Homepage card** — `docs_new/cookbook/<category>/intro.mdx`: if the org already has a
|
||||
`<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
|
||||
(title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
|
||||
ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
|
||||
transparent** PNG → `docs_new/cards/logos/<org-slug>.png` per
|
||||
[references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
|
||||
is gitignored repo-wide). Never invent or copy a logo.
|
||||
|
||||
## Phase 3 — Validate
|
||||
|
||||
```bash
|
||||
cd docs_new
|
||||
mint validate # frontmatter, missing nav entries, MDX/JSX errors
|
||||
mint broken-links
|
||||
mint dev # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
|
||||
```
|
||||
|
||||
Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
|
||||
Playground inherits the Deploy selection live; each axis toggle produces the expected
|
||||
diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
|
||||
hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
|
||||
and is gone from same-vendor siblings; the homepage card points to the new model.
|
||||
|
||||
## Phase 4 — Interactive testing
|
||||
|
||||
The user deploys each cell, runs the benches, and pastes results; you fill the data:
|
||||
- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
|
||||
- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
|
||||
speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
|
||||
`0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
|
||||
cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
|
||||
|
||||
## Phase 5 — Prose & config tips
|
||||
|
||||
**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
|
||||
carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
|
||||
rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
|
||||
user notes: §1 Model Introduction (description, links, params, license, variants table),
|
||||
§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
|
||||
Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
|
||||
parser's output shape; each runnable block gets an `**Output Example:**`).
|
||||
|
||||
## Phase 6 — Review
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Git workflow
|
||||
|
||||
Always branch — never commit to main directly.
|
||||
|
||||
```bash
|
||||
git checkout -b add-<model>-cookbook
|
||||
git add docs_new/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs_new/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
|
||||
docs_new/cookbook/<category>/<Vendor>/<Model>.mdx \
|
||||
docs_new/docs.json docs_new/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
@@ -0,0 +1,220 @@
|
||||
# Cookbook config reference (fields · cells · playground · MDX)
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill. This is the field-by-field
|
||||
contract for when the clone needs more than a rename. The two engine files are
|
||||
the canonical specs — read their headers first:
|
||||
|
||||
- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the 5-dim matrix widget; lists every config field.
|
||||
- [`_playground.jsx`](../../../../docs_new/src/snippets/_playground.jsx) — the diff-based override widget; lists the `playgroundFeatures` axes + the `AXIS_HANDLERS` interface.
|
||||
|
||||
Engine extension (adding a new playground axis) lives in [engine-axis.md](engine-axis.md).
|
||||
|
||||
---
|
||||
|
||||
## 2.1 Create the config file
|
||||
|
||||
**Path**: `docs_new/src/snippets/configs/<vendor>/<model>.jsx`. The vendor folder is
|
||||
the HuggingFace org (`deepseek-ai`, `Qwen`, `moonshotai`, ...); the file
|
||||
name is a short hyphenated model id (`deepseek-v4`, `qwen3.5`, ...).
|
||||
|
||||
**Shape**: must be a single `export const config = { ... }` literal. Do not
|
||||
use function calls, spreads, fragment refs, or IIFE — Mintlify re-evaluates
|
||||
this export at hydration time with module-level identifiers out of scope,
|
||||
and any non-literal value crashes with `ReferenceError`.
|
||||
|
||||
**Required fields** (engine reads these — see the `_deployment.jsx` header for
|
||||
the full contract):
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
|
||||
| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
|
||||
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
|
||||
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
|
||||
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Common ids: `low-latency`, `balanced`, `high-throughput`. |
|
||||
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
|
||||
| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. |
|
||||
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
|
||||
| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
|
||||
| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
|
||||
|
||||
**Optional fields**:
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
|
||||
| `dockerImages` | `{[hwId]: string}` | Per-hw image name for `docker run` framing. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `ACCURACY_LABELS` + `benchmarkCommands.accuracy`. |
|
||||
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the value that matches the `model` dropdown in your issue template so it's pre-selected when the issue opens. |
|
||||
|
||||
## 2.2 Author the 5-dim matrix (`cells[]`)
|
||||
|
||||
Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
|
||||
```js
|
||||
{
|
||||
match: { hw: "b200", variant: "flash", quant: "fp4",
|
||||
strategy: "low-latency", nodes: "single" },
|
||||
verified: true, // green "Verified" badge; absence = yellow
|
||||
env: [
|
||||
"SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024",
|
||||
],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}", // {{MODEL_NAME}} resolves from modelNames
|
||||
"--tp 4",
|
||||
"--moe-runner-backend flashinfer_mxfp4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
|
||||
- `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`,
|
||||
`strategy`, `nodes`. The engine looks up cells by tuple equality.
|
||||
- `env` and `flags` are FLAT literals. The engine does NOT expand
|
||||
fragments, aliases, or templates — it consumes them verbatim
|
||||
(only `{{PLACEHOLDER}}` substitutions happen at render time).
|
||||
- DO NOT include `--nnodes` / `--node-rank` / `--dist-init-addr` in
|
||||
`cell.flags` for multi-node cells. The renderer injects them
|
||||
automatically from `match.nodes` (`multi-N` → N nodes).
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
|
||||
**Cells are denormalized on purpose** — common flags repeat across cells.
|
||||
This makes each cell self-contained and easy to verify. When sweeping a
|
||||
common change, edit every cell.
|
||||
|
||||
**Avoid premature cells**: only add a cell for a (hw × variant × quant ×
|
||||
strategy × nodes) combination if you have a recipe that has been tested or
|
||||
at least sanity-checked. The engine greys out un-listed combinations
|
||||
automatically.
|
||||
|
||||
## 2.3 Configure `playgroundFeatures` (optional)
|
||||
|
||||
The Playground widget is opt-in per axis. Add only the axes that make sense
|
||||
for this model. Recognised axis keys and their schemas (full reference in
|
||||
the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select + EP knob | Model is MoE and supports multiple `--moe-*-backend` choices. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
|
||||
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
|
||||
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
|
||||
| `megamoe` | Single-select with hw/strategy gating | Blackwell-only kernel fusion variant. |
|
||||
|
||||
**Per-chip constraints**: any chip entry in any axis can be wrapped with
|
||||
`hide` / `disable` constraint objects:
|
||||
|
||||
```js
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }
|
||||
```
|
||||
|
||||
- `hide` — chip omitted entirely (use for hard impossibilities).
|
||||
- `disable` — chip greyed out with tooltip (soft warning).
|
||||
- Constraints are AND across keys, OR within each key's array.
|
||||
- Bare `disabled: true` / `disable: true` is a static always-disabled form
|
||||
(used for "Coming soon" chips).
|
||||
|
||||
## 2.4 Create the MDX page
|
||||
|
||||
Path: `docs_new/cookbook/<category>/<Vendor>/<Model>.mdx`. Import both widgets and
|
||||
the per-model config, render them inside the relevant sections:
|
||||
|
||||
```mdx
|
||||
## Deployment
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/<vendor>/<model>.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/<vendor>/<model>-benchmarks.jsx";
|
||||
|
||||
{/* Install is a PREREQUISITE — keep it compact + collapsed at the top of the
|
||||
Deploy section (NOT a numbered section). Tabs mirror the widget's
|
||||
Python/Docker toggle. */}
|
||||
<a id="install" />
|
||||
<Accordion title="Install SGLang">
|
||||
<Tabs>
|
||||
<Tab title="Python (pip / uv)">…pip / uv install…</Tab>
|
||||
<Tab title="Docker">…docker pull + a `docker run … sglang serve` example…</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
[model-specific tuning notes, caveats, links]
|
||||
|
||||
## Playground
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
```
|
||||
|
||||
**Heading slugs matter** — the two widgets cross-link by scrolling to each
|
||||
other's section id (Mintlify auto-slugs headings: lowercase, spaces →
|
||||
hyphens, punctuation dropped). The engines look up:
|
||||
|
||||
- the **Deploy** panel by id `deployment` (falls back to `deploy`) — used
|
||||
by the Playground's "↑ Switch base" button and by deep-link scroll-on-
|
||||
load. Title the section `## Deployment` (or `## Deploy`).
|
||||
- the **Playground** by id `playground` — used by `_deployment.jsx`'s
|
||||
"Open the Playground →" link. Title the section `## Playground`.
|
||||
|
||||
Avoid numbered headings like `## 3. Model Deployment` (slug
|
||||
`3-model-deployment`) for these two sections — the cross-links would break.
|
||||
The Playground reads the Deploy selection live via the URL hash + the
|
||||
`sglang-deploy-sel` custom event, so the two can live in different parts
|
||||
of the page.
|
||||
|
||||
The `benchmarks` prop is **optional**. It points at a sibling
|
||||
`<model>-benchmarks.jsx` file (one entry per cell, keyed by the same
|
||||
`match` tuple) that renders an accuracy + speed sub-card under the command
|
||||
box; omit the import and the prop if the cookbook has no measured numbers
|
||||
yet. See the `_deployment.jsx` header and `deepseek-v4-benchmarks.jsx` for
|
||||
the full speed/accuracy schema.
|
||||
|
||||
To let users *reproduce* those numbers, add a `benchmarkCommands` block to
|
||||
the config (§2.1, next to `curl`). When present alongside `benchmarks`, the
|
||||
benchmark card grows a **"⚡ Reproduce"** button that opens a modal listing
|
||||
the runnable commands for the current cell — one `bench_serving` command for
|
||||
Speed (with concurrency chips that rewrite `--max-concurrency`) plus an
|
||||
Accuracy command with a chip per eval. No separate benchmark section needed.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls (authoring)
|
||||
|
||||
**Stale URL hash hydration** — If a user shares a link from an old cell
|
||||
catalog and the hash names an impossible combination, `_deployment.jsx`'s
|
||||
`validateSelection` snaps to the nearest real cell. The Playground reads
|
||||
the hash too — make sure cookbook removals don't leave dangling shared
|
||||
links pointing at hardware/quant combos that no longer exist.
|
||||
|
||||
**Mintlify constraints** — Module-level statements are stripped. The config
|
||||
MUST be a single `export const config = { ... }` literal — no function calls,
|
||||
spreads, fragment refs, or IIFE (Mintlify re-evaluates the export at hydration
|
||||
with module-level identifiers out of scope; any non-literal crashes with
|
||||
`ReferenceError`). In MDX, capitalized JSX tags get rebound — use the built-in
|
||||
Mintlify components (`<Accordion>`, `<Tabs>`, `<Card>`, ...) as documented.
|
||||
Avoid `!(x in y)` anywhere (Mintlify's AST walker crashes on it) — use
|
||||
`obj.key === undefined`.
|
||||
|
||||
**Per-cell denormalization** — Cells repeat common flags on purpose. Do
|
||||
not factor them into a shared `commonFlags` array — Mintlify will fail
|
||||
to inline the reference. If you need to sweep a flag across cells, do it
|
||||
with a global find-replace in the config file.
|
||||
@@ -0,0 +1,242 @@
|
||||
# Engine extension: add a new playground feature axis
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill. **Rare** — adding a model
|
||||
cookbook is data-only and never needs this. The current 8 built-in axes
|
||||
(`attention`, `moe`, `parsers`, `speculative`, `pdDisagg`, `hicache`,
|
||||
`hisparse`, `megamoe`) already cover the SGLang feature surface most cookbooks
|
||||
need. Only add a new axis if a real cookbook needs it and the feature does not
|
||||
fit any existing axis. Touches `_playground.jsx` only.
|
||||
|
||||
For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
|
||||
|
||||
---
|
||||
|
||||
## 3.1 Decide
|
||||
|
||||
Before touching the engine, confirm:
|
||||
|
||||
- The feature is a STABLE part of the SGLang CLI surface (will appear in
|
||||
multiple cookbooks, not one-off).
|
||||
- The feature cannot be expressed as a new option inside an existing axis
|
||||
(e.g. a new MoE backend belongs in `moe.backend.options`, not a new
|
||||
axis).
|
||||
- The feature has a clean strip-prefix → emit-flag pattern.
|
||||
|
||||
If unsure, add it as data first (in one cookbook's config under an
|
||||
existing axis) before promoting it to a built-in axis.
|
||||
|
||||
## 3.2 Pick the axis id and state shape
|
||||
|
||||
The axis id is the key in both `config.playgroundFeatures` and the
|
||||
internal `deltas` object. Use camelCase, descriptive but short:
|
||||
`mambaCache`, `attentionBackend`, `kvCacheDtype`.
|
||||
|
||||
The state shape is whatever `initState` returns. Common shapes:
|
||||
|
||||
- Single-select: a string sentinel (e.g. `"disabled"` / `"current"` / an
|
||||
option id).
|
||||
- Multi-toggle: `{[itemId]: bool}`.
|
||||
- Sub-knobs: `{[knobId]: value | null}`.
|
||||
- Compound (axis with its own internal sub-state, like PD-Disagg's
|
||||
`{mode, ibDevice}`): a plain object.
|
||||
|
||||
Pick ONE "inherit base" sentinel and document it in the handler comment.
|
||||
|
||||
## 3.3 Implement the handler
|
||||
|
||||
Add one entry to `AXIS_HANDLERS` in `_playground.jsx`.
|
||||
The handler owns everything: state init, apply (strip+insert), hidden-revert,
|
||||
AND the JSX render. Engine main loop iterates `AXIS_HANDLERS` and calls each
|
||||
method by name — adding a new axis is genuinely a one-place change.
|
||||
|
||||
Template:
|
||||
|
||||
```js
|
||||
// ---- Axis: <Title> ----------------------------------------------------
|
||||
// <one-paragraph description of what this axis controls and why it
|
||||
// exists. Mention the SGLang feature it wraps and the strip/insert
|
||||
// policy.>
|
||||
<axisId>: {
|
||||
initState: (fc) => /* initial state value */,
|
||||
|
||||
// Called when base cell changes. Return new value if the picked option
|
||||
// is now hidden by a constraint; otherwise return value unchanged.
|
||||
// Disabled picks are intentionally NOT auto-reverted (soft warning).
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
// ... return value or a new value
|
||||
return value;
|
||||
},
|
||||
|
||||
// Pure function. Receives the current (flags, env) and returns the next
|
||||
// (flags, env). Do NOT mutate inputs. The `value` argument is whatever
|
||||
// initState returned. The `fc` argument is config.playgroundFeatures[axisId].
|
||||
// The `sel` argument is the current base cell selection. The `h`
|
||||
// argument is the helpers bundle (strip/insert primitives + anchors).
|
||||
apply: ({ flags, env, value, fc, sel, h, derived }) => {
|
||||
if (/* value is the inherit-base sentinel */) return { flags, env };
|
||||
flags = h.stripFlagsByFirstToken(flags, [/* prefixes this axis owns */]);
|
||||
if (/* an option is picked */) {
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_<X>, [/* new flags */]);
|
||||
// or: flags = h.insertBeforeTail(flags, [/* new flags */]);
|
||||
// if the axis mutates env:
|
||||
// env = h.stripEnvByPrefix(env, fc.stripEnv || []);
|
||||
// env = [...env, /* additional env vars */];
|
||||
}
|
||||
return { flags, env };
|
||||
},
|
||||
|
||||
// Optional: read the base cell's flag array back into the same shape
|
||||
// initState/apply use. Render shows this as the default selection
|
||||
// (dropdown option or checked chip) when the state slot is the inherit
|
||||
// sentinel — so the user sees the cell's actual --tp / MoE backend /
|
||||
// spec preset instead of an opaque "Auto." When derive returns a real
|
||||
// value, the inherit-sentinel option is hidden from the control. Apply
|
||||
// also receives the derived
|
||||
// value (as `derived`) and may use it as a no-op shortcut when the
|
||||
// user's pick matches base. Skip when your axis owns flags that never
|
||||
// appear in base cells (PD-Disagg / HiCache / MegaMoE).
|
||||
// deriveFromBase: (cell, fc, h) => ({ ... }) | null,
|
||||
|
||||
// Optional: hints for the renderer. Currently only pdDisagg uses this
|
||||
// to report its role banner. Omit if not needed.
|
||||
// getRenderHints: (value, fc) => ({ pdMode: ... }) | null,
|
||||
|
||||
// Returns the axis card JSX. The outer div MUST have key={axisId} so
|
||||
// React can track it in the engine's map loop. Return null for
|
||||
// axis-level gating (e.g. MegaMoE on Hopper). Lay out as a single
|
||||
// compact horizontal row: title on the left, fields after.
|
||||
render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => {
|
||||
if (/* axis-level gating fails */) return null;
|
||||
return (
|
||||
<div key={axisId} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
<span style={s.axisTitle}>Axis Title</span>
|
||||
{/* For multi-option fields, use renderSelect(...) — the default.
|
||||
For on/off toggles or single-select chip groups, use
|
||||
renderChip instead (see "Control choice" in the conventions
|
||||
below). Read state from `value`; write via `setValue(next)`
|
||||
(replaces the whole axis slot). */}
|
||||
<span style={s.field}>
|
||||
<span style={s.fieldLabel}>Field</span>
|
||||
{renderSelect(value.slot, fc.entries, (v) =>
|
||||
setValue({ ...value, slot: v }), base)}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**Important conventions**:
|
||||
|
||||
- Insert the entry in the position you want it rendered. `AXIS_HANDLERS`
|
||||
is iterated in insertion order for both render and apply.
|
||||
- Use `h.ANCHOR_NEAR_*` constants for insertion. Add a new anchor to the
|
||||
helpers bundle if your axis needs to land somewhere new in the flag
|
||||
block.
|
||||
- Use lowercase HTML JSX tags only. Capitalized tags get rebound by
|
||||
Mintlify.
|
||||
- Inside `render`, read state via `value` (the slice for this axis).
|
||||
Write state via `setValue(next)` (replaces the whole slice). For
|
||||
compound axes, do `setValue({ ...value, [k]: nextK })`.
|
||||
- Layout: one `s.compactRow` per axis card, `s.axisTitle` for the
|
||||
leading label, one `s.field` per (label + input) pair.
|
||||
- Control choice — `renderSelect` vs `renderChip`:
|
||||
- `renderSelect(current, entries, onPick, base, labelFor?, opts?)` is
|
||||
the **default** compact control (a `<select>` dropdown). It filters
|
||||
hidden chips and disables greyed-out ones internally — no per-chip
|
||||
`evaluateChip` loop needed in the render body. Most axes use it
|
||||
(attention, moe, pdDisagg, hisparse, hicache, megamoe). Pass
|
||||
`{ hideValues: [<sentinel>] }` when your `deriveFromBase` resolved to
|
||||
a real value, so the inherit-sentinel ("Auto" / "Inherited" /
|
||||
"current") doesn't clutter the dropdown.
|
||||
- `renderChip(label, current, value, onPick, { disabled?, disabledReason? })`
|
||||
renders a **button** instead of a dropdown row. Use it for a chip
|
||||
group when you want the options laid out as buttons. It serves two
|
||||
shapes:
|
||||
- **Multi-toggle** (Parsers) — one independent on/off chip per item;
|
||||
`current` is that item's effective bool, `value` is `true`, so the
|
||||
chip is "checked" when the item is on.
|
||||
- **Single-select** (Speculative) — a radio-style group; pass the
|
||||
group's effective value as `current` and each option's id as
|
||||
`value`, so exactly one chip is checked (`current === value`).
|
||||
Chip groups own their visibility/disable filtering: loop
|
||||
`h.evaluateChip(opt, base)` in the render body, skip `c.hidden`,
|
||||
filter the inherit-sentinel yourself when `deriveFromBase` resolved
|
||||
to a real value, and forward `c.disabled` / `c.disableReason` into
|
||||
`renderChip`'s opts (this is what surfaces a disabled chip's tooltip,
|
||||
e.g. a "Coming soon" entry).
|
||||
- Selected chips use the same terracotta (`#D45D44`) as the Deploy
|
||||
panel's selected button, so both widgets read as one
|
||||
visual system. Don't introduce a per-axis accent color.
|
||||
- Default-from-base: if your axis can be read out of base cells'
|
||||
flags, implement `deriveFromBase` and have your render show the
|
||||
derived value when state is the sentinel (e.g.
|
||||
`const eff = value.tp !== null ? value.tp : (derived && derived.tp)`).
|
||||
This is what makes a fresh playground load show the user's actual
|
||||
recipe instead of "auto." Flag-parsing helpers on `h`:
|
||||
`parseIntFlag`, `hasFlag`, `findFlagArg`.
|
||||
- **Avoid the `in` operator wrapped in unary** (`!(x in y)`). Mintlify's
|
||||
AST walker crashes on it (`TypeError: this[e] is not a function`). Use
|
||||
`obj.key === undefined` or `obj.id !== undefined` instead. Bare
|
||||
`if (key in obj)` (no surrounding `!`) is fine.
|
||||
|
||||
## 3.4 Document the per-cookbook schema
|
||||
|
||||
Edit the file header in `_playground.jsx` to add your new axis to the
|
||||
"Recognised keys" list, with a one-line description of its schema.
|
||||
Optionally add a paragraph below explaining its strip/insert policy.
|
||||
|
||||
Update the §2.3 axis table in [authoring-reference.md](authoring-reference.md) to list the new axis.
|
||||
|
||||
## 3.5 Migrate cookbooks that need it
|
||||
|
||||
For each cookbook that should expose this axis, add a
|
||||
`playgroundFeatures.<axisId>` entry to its config. Verify the chip group
|
||||
renders, options apply correctly, and the diff matches expectations.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls (engine work)
|
||||
|
||||
**Insertion anchor misses** — `insertAfter` falls back to right-after
|
||||
`--model-path` if none of its anchor prefixes are present. If your axis
|
||||
emits flags that should land somewhere specific, include the most likely
|
||||
anchor prefixes in your call. Order doesn't matter (set semantics).
|
||||
|
||||
**Conditional strips** — Some axes strip ONLY when overridden
|
||||
(`attention.tp`, `moe.backend`, `speculative`, `megamoe`). Others strip
|
||||
UNCONDITIONALLY whenever declared (`parsers`, `pdDisagg`, `hicache`). The
|
||||
header comment in `AXIS_HANDLERS` documents which policy each axis uses;
|
||||
follow the same pattern when adding a new axis. If unsure, prefer
|
||||
conditional strip — it preserves base behavior when the user does not
|
||||
opt in.
|
||||
|
||||
**Closure of `AXIS_HANDLERS`** — Inside a handler method, you can
|
||||
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (megamoe
|
||||
does this for `_gateOpen`). This works because `AXIS_HANDLERS` is in
|
||||
lexical scope. Do NOT use this for general logic — it tightly couples
|
||||
handlers. Reserve it for one handler's helpers shared between its own
|
||||
`render` and `revertHidden`.
|
||||
|
||||
---
|
||||
|
||||
## Review checklist for a new-axis PR
|
||||
|
||||
- [ ] `AXIS_HANDLERS` is the ONLY place that mentions the new axis id
|
||||
(apart from per-cookbook config). No `if (axisId === '<new>')`
|
||||
branches anywhere in the engine.
|
||||
- [ ] `initState` is deterministic and idempotent (does not depend on
|
||||
the base cell).
|
||||
- [ ] `apply` is pure — does not mutate inputs.
|
||||
- [ ] `revertHidden` returns the same reference when nothing changed
|
||||
(avoids unnecessary re-renders).
|
||||
- [ ] `render` returns `null` when axis-level gating fails (whole card
|
||||
hidden) — does not render an empty placeholder.
|
||||
- [ ] `render` sets `key={axisId}` on its outer element.
|
||||
- [ ] No `!(x in y)` patterns introduced (Mintlify AST walker crashes).
|
||||
- [ ] File header lists the new axis in "Recognised keys".
|
||||
- [ ] The §2.3 table in `authoring-reference.md` lists the new axis.
|
||||
- [ ] One existing cookbook config is updated to consume the new axis,
|
||||
and visual verification shows the diff is correct.
|
||||
@@ -0,0 +1,107 @@
|
||||
# MDX authoring rules (Mintlify) + invocation-example patterns
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill (Phase 5, writing the page prose).
|
||||
These are model-agnostic Mintlify hygiene rules — the most common review findings.
|
||||
The cookbook is **Mintlify**, not Docusaurus.
|
||||
|
||||
## Mintlify syntax
|
||||
|
||||
**Allowed components**: `<Card>`, `<CardGroup>`, `<Note>`, `<Tip>`, `<Warning>`,
|
||||
`<Info>`, `<Accordion>`, `<AccordionGroup>`, `<Steps>`, `<Step>`, `<Tabs>`, `<Tab>`,
|
||||
`<CodeGroup>`, `<Frame>`, `<Icon>`.
|
||||
|
||||
**Forbidden** (flag every occurrence):
|
||||
- Docusaurus admonitions (`:::note` / `:::warning` / …) — use `<Note>` / `<Warning>`.
|
||||
- `@site/...` / `@theme/...` imports — use absolute `/src/snippets/...`.
|
||||
- GitHub alert blocks (`> [!NOTE]`, `> [!WARNING]`).
|
||||
- **Markdown pipe tables** on new pages — use JSX `<table>` (see below).
|
||||
- Inline `<details>` / `<summary>` — use `<Accordion>`.
|
||||
- Unknown / non-Mintlify components.
|
||||
- `<CardGroup>` / `<Card>` on individual model pages — those are for category `intro.mdx` only.
|
||||
|
||||
**Code fences**: always labeled — ` ```python Example `, ` ```bash Command `,
|
||||
` ```shell Command `, ` ```text Output `. When nesting a fenced block inside another,
|
||||
the **outer** fence uses four backticks.
|
||||
|
||||
**Internal links**: root-relative, no extension (`/cookbook/<category>/<Vendor>/<Model>`);
|
||||
`docs.sglang.io` is canonical. Flag `.md`/`.mdx` extensions and `../`-relative page links
|
||||
in body prose. (Existing cookbook pages do use `../../../docs/...` for cross-links into
|
||||
the non-cookbook docs tree — that's the established exception; don't introduce new ones.)
|
||||
|
||||
## JSX tables (required for all tables on new pages)
|
||||
|
||||
```jsx
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Col</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "9px 12px"}}>cell</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
```
|
||||
|
||||
Alternate column background colors (`rgba(255,255,255,0.02)` / `0.05`) for readability;
|
||||
adjust `<colgroup>` widths for 3- or 5-column tables. The DeepSeek-V4 page §1 variants
|
||||
table is a live reference.
|
||||
|
||||
## Invocation-example patterns (§3 Advanced Usage)
|
||||
|
||||
- **Reasoning-parser output shape must match the example**:
|
||||
- *Separate-field* parsers (most qwen/glm, `kimi_k2`, `deepseek-v4`): thinking lands
|
||||
in `message.reasoning_content`, answer in `message.content` — print both.
|
||||
- *Inline-tag* parsers (e.g. `minimax-append-think`): thinking is wrapped in
|
||||
`<think>...</think>` **inside** `message.content` — the client parses the tags; for
|
||||
streaming, buffer and split on the markers.
|
||||
Pick the pattern from the model card / SGLang docs for that specific parser.
|
||||
- **Hybrid reasoning models**: show both thinking-on (default) and thinking-off
|
||||
(`extra_body={"chat_template_kwargs": {"thinking": False}}` or `enable_thinking: False`).
|
||||
- **Tool-call follow-up on thinking models**: the final assistant turn may put text in
|
||||
`reasoning_content` instead of (or with) `content` — print both so the output isn't a
|
||||
misleading `None`.
|
||||
- **Every runnable block** is immediately followed by `**Output Example:**` + a
|
||||
` ```text Output ` block with **real** server output (verbatim, not paraphrased).
|
||||
`Pending update...` is acceptable only with the user's explicit acknowledgement.
|
||||
- **Do not hardcode sampling params** (`temperature`, `top_p`) in sample code — SGLang
|
||||
uses `generation_config.json` defaults. Listing "Recommended generation" in §1 is fine.
|
||||
- Format raw API objects (`ChatCompletionMessage(...)`) into readable Reasoning /
|
||||
Content / Tool Calls blocks.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
- **Top-level `description:`** is the canonical field — it sets the page's SEO meta
|
||||
description (`og:`/`twitter:description` fall back to it) AND renders as the visible
|
||||
**subtitle** under the title, filling the header band before the first heading. Give every
|
||||
page a one-line top-level `description` (a lede / value prop) — without it, a page that
|
||||
opens straight into `## Deployment` shows an empty gap under the title (the title and
|
||||
`## Deployment` are the same size, so they read as two bare headings). Do **not** put the
|
||||
description inside a `metatags` block — `metatags` is for other/custom tags, and
|
||||
`metatags.description` is redundant with (and non-canonical vs) the top-level field.
|
||||
- **Write it for SEO** (it doubles as the search-result snippet): front-load the exact
|
||||
model name + intent — e.g. `Deploy <Model> with SGLang — …` — aim for ~150–160 chars, and
|
||||
pack secondary keywords (variants + sizes, `Mixture-of-Experts` / architecture, target
|
||||
GPUs). Phrase it as a value prop, not a generic "`<Model>` is a … model" intro.
|
||||
- Frontmatter MUST be the first thing in the file — no comment or blank line before the
|
||||
opening `---`.
|
||||
|
||||
## Commands & ports
|
||||
|
||||
- **Deploy/launch** commands use `sglang serve --model-path …` — never
|
||||
`python -m sglang.launch_server` / `python3 -m sglang.launch_server` (deprecated).
|
||||
- **Benchmark workload** commands use `python3 -m sglang.bench_serving …` (never bare
|
||||
`python -m`); built-in accuracy scripts use `python3 benchmark/...`.
|
||||
- Port **30000** everywhere on a page — launch, curl, client `base_url`, and bench must
|
||||
agree. Keep one canonical deploy command (the Deploy widget) and don't re-paste launch
|
||||
commands across sections; the documented command must match the widget's output for the
|
||||
same selection (doc ↔ config parity).
|
||||
|
||||
## Factual hygiene
|
||||
|
||||
- License must match the actual HuggingFace license (don't copy from another model).
|
||||
- HF URLs resolve to a real model; Docker images from `lmsysorg/sglang`.
|
||||
- No Google-Drive image links (they don't render); host images in the repo.
|
||||
- Shell placeholders are `export VAR=<value>`, not `export VAR=${VAR}` (a bash no-op).
|
||||
- `tag: NEW` is sparing — at most one per `<category>/<Vendor>/` dir (the newest); strip
|
||||
it from siblings when adding a new NEW page.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Vendor card logo (new brand only)
|
||||
|
||||
A new vendor/brand in the cookbook landing grid needs a card logo at
|
||||
`docs_new/cards/logos/<org-slug>.png`. **Ask the user for the brand's logo, then generate
|
||||
the conforming PNG** — never invent, copy, or hallucinate one, and never ship a
|
||||
non-conforming file. Reference: PR #27400 (added `tencent.png` + `poolside.png`).
|
||||
|
||||
If the org already has a card/logo, do nothing here — only update the `<Card href>`.
|
||||
|
||||
## Spec (match the existing logos exactly)
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Path | `docs_new/cards/logos/<org-slug>.png` — lowercase, matches the `img=` in the `<Card>` |
|
||||
| Canvas | **940 × 525** px |
|
||||
| Mode | **RGBA**, fully **transparent** background |
|
||||
| Content | **Icon-only** — the brand glyph/mark (the "swirl"), **no wordmark text** |
|
||||
| Placement | centered; glyph ≈ 0.33 × width and ≈ 0.50 × height of the canvas |
|
||||
|
||||
Why icon-only + transparent: cards render on both light and dark backgrounds, so a baked-in
|
||||
(usually black) wordmark vanishes on dark. `deepseek.png` / `ernie.png` are 940×525 RGBA
|
||||
exemplars — eyeball your output against them.
|
||||
|
||||
## 1. Get the source
|
||||
|
||||
Ask the user for the brand logo (SVG preferred → crisp + already transparent; else a high-res
|
||||
transparent PNG, or a link to the official press/brand asset). Prefer an **icon-only** source;
|
||||
if they only have a full lockup, ask them to crop the glyph, or crop it yourself.
|
||||
|
||||
If the user **pasted** an image inline, it may not be on disk — recover the base64 `image`
|
||||
block from the session transcript (`~/.claude/projects/<slug>/*.jsonl`) and decode it to a file.
|
||||
|
||||
## 2. Generate (Pillow)
|
||||
|
||||
There's no system Pillow — use a venv:
|
||||
|
||||
```bash
|
||||
python3 -m venv /tmp/logo-venv && /tmp/logo-venv/bin/pip install -q Pillow
|
||||
```
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
src = Image.open("SOURCE").convert("RGBA") # icon-only, already transparent
|
||||
W, H = 940, 525
|
||||
target_h = round(H * 0.50) # glyph ≈ half the canvas height
|
||||
scale = target_h / src.height
|
||||
glyph = src.resize((round(src.width * scale), target_h), Image.LANCZOS)
|
||||
canvas = Image.new("RGBA", (W, H), (0, 0, 0, 0)) # transparent
|
||||
canvas.paste(glyph, ((W - glyph.width) // 2, (H - glyph.height) // 2), glyph)
|
||||
canvas.save("docs_new/cards/logos/<org-slug>.png")
|
||||
```
|
||||
|
||||
Notes:
|
||||
- **Wordmark present?** Crop to the glyph first (or ask the user for an icon-only asset). Don't
|
||||
ship text in the logo.
|
||||
- **Solid background?** Don't auto-strip it (risky) — ask the user for a transparent source.
|
||||
- **SVG source?** Rasterize at high res first (`cairosvg` / `rsvg-convert`), then run the above.
|
||||
- If the glyph is much wider than tall, cap by width instead (≈ 0.33 × W) so it doesn't overflow.
|
||||
|
||||
## 3. Verify
|
||||
|
||||
```bash
|
||||
sips -g pixelWidth -g pixelHeight -g hasAlpha docs_new/cards/logos/<org-slug>.png
|
||||
# → pixelWidth: 940 pixelHeight: 525 hasAlpha: yes
|
||||
```
|
||||
|
||||
## 4. Wire + track + validate
|
||||
|
||||
```bash
|
||||
# Card in the landing grid (keep card order aligned with the docs.json nav order):
|
||||
# <Card title="<NavGroup>" mode="card"
|
||||
# href="/cookbook/<category>/<Vendor>/<Model>"
|
||||
# img="/cards/logos/<org-slug>.png" />
|
||||
git add -f docs_new/cards/logos/<org-slug>.png # root .gitignore ignores *.png repo-wide
|
||||
cd docs_new && mint validate && mint broken-links # confirms the card href + img resolve
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
|
||||
// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>-benchmarks.jsx and
|
||||
// fill measured numbers — OR delete this file entirely if you have none yet (the
|
||||
// MDX simply omits the `benchmarks` import/prop).
|
||||
//
|
||||
// One entry per cell `match` tuple (same 5 keys as config cells). The card stays
|
||||
// "pending" until an entry has a non-null speed metric or accuracy. Speed shape:
|
||||
// speed: [{ workload: {dataset, isl, osl, max_concurrency}, ttft_ms, tpot_ms,
|
||||
// tokens_per_sec_per_gpu }, ...] // interactivity is derived (1000/TPOT)
|
||||
// Per-cell `accuracy: { <key>: <pct> }` overrides the config's defaultAccuracy.
|
||||
|
||||
export const benchmarks = [
|
||||
// EXAMPLE — one filled entry showing the shape; replace numbers, add one per cell.
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.0.0", // TODO: ASK the user for the sglang version these numbers were measured on — don't invent one
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
|
||||
],
|
||||
},
|
||||
// Bare-match stubs (no data yet) are fine — the card shows "pending" for these.
|
||||
{ match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "h100", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
{ match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" } },
|
||||
];
|
||||
@@ -0,0 +1,374 @@
|
||||
// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
|
||||
// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx, then:
|
||||
// 1. replace every __TOKEN__,
|
||||
// 2. fill cells[] with your verified recipes (the examples below show the shape),
|
||||
// 3. DELETE the hardware / playground axes / quantizations your model lacks.
|
||||
//
|
||||
// Instantiation tokens (skill fills these; distinct from the engine's runtime
|
||||
// {{PLACEHOLDER}} which MUST survive verbatim into the output):
|
||||
// __MODEL_DISPLAY__ display name, e.g. "DeepSeek-V4"
|
||||
// __MODEL_SLUG__ file slug, e.g. "deepseek-v4"
|
||||
// __HF_ORG__ HuggingFace org, e.g. "deepseek-ai"
|
||||
// __HF_REPO__ HuggingFace repo, e.g. "DeepSeek-V4-Flash"
|
||||
// __REASONING_PARSER__ e.g. "deepseek-v4" (delete the parsers axis if none)
|
||||
// __TOOLCALL_PARSER__ e.g. "deepseekv4" (delete the parsers axis if none)
|
||||
//
|
||||
// Mintlify: single `export const config = {...}` literal — no spreads/calls/IIFE,
|
||||
// no `!(x in y)`. Cells are denormalized: no --nnodes/--node-rank/--dist-init-addr/
|
||||
// --host/--port literals (the engine injects them).
|
||||
|
||||
export const config = {
|
||||
modelName: "__MODEL_DISPLAY__",
|
||||
|
||||
// List ONLY hardware you ship a cell for; unlisted ids auto-grey-out. The full
|
||||
// catalog is below — delete the families your model doesn't support (e.g. drop
|
||||
// every `mi*` if there's no AMD recipe).
|
||||
supportedHardware: [
|
||||
"h100", "h200", "b200", "b300", "gb200", "gb300",
|
||||
"mi300x", "mi325x", "mi350x", "mi355x",
|
||||
],
|
||||
|
||||
// OPTIONAL — declare GPUs the shared HARDWARE_CATALOG (in _deployment.jsx) doesn't
|
||||
// carry (workstation / desktop / future chips). The engine merges these in, so a
|
||||
// model-specific GPU is config data, never an engine-catalog edit. Add the id to
|
||||
// supportedHardware above too. Delete if you only use catalog GPUs.
|
||||
// hardware: [
|
||||
// { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" },
|
||||
// ],
|
||||
|
||||
// 2nd dim. Single-element `default` if the model has no variant axis; else list
|
||||
// real variants (e.g. {id:"flash",...},{id:"pro",...}) and key modelNames/
|
||||
// defaultAccuracy by them.
|
||||
variants: [
|
||||
{ id: "default", label: "Default" },
|
||||
],
|
||||
// 3rd dim. Keep only what your model ships (BF16 / FP8 / FP4 / …).
|
||||
quantizations: [
|
||||
{ id: "bf16", label: "BF16" },
|
||||
{ id: "fp8", label: "FP8" },
|
||||
{ id: "fp4", label: "FP4" },
|
||||
],
|
||||
strategies: [
|
||||
{ id: "low-latency", label: "Low-Latency" },
|
||||
{ id: "balanced", label: "Balanced" },
|
||||
{ id: "high-throughput", label: "High-Throughput" },
|
||||
],
|
||||
// `multi-N` id carries the node count for `--nnodes N`.
|
||||
nodesOptions: [
|
||||
{ id: "single", label: "Single Node" },
|
||||
{ id: "multi-2", label: "Multi-Nodes" },
|
||||
],
|
||||
|
||||
// HF slug lookup. Key by `variant|quant` (or `hw|variant|quant` for a per-hw
|
||||
// repackaging, e.g. an FP8 conversion only valid on one platform).
|
||||
modelNames: {
|
||||
"default|bf16": "__HF_ORG__/__HF_REPO__",
|
||||
"default|fp8": "__HF_ORG__/__HF_REPO__",
|
||||
"default|fp4": "__HF_ORG__/__HF_REPO__",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
NODE0_IP: { target: "command", label: "Head node IP", default: "<node0-ip>" },
|
||||
NODE_RANK: { target: "command", label: "This node rank", default: "<node-rank>" },
|
||||
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
|
||||
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||
|
||||
// OPTIONAL — powers the benchmark card's "⚡ Reproduce" modal. Delete the whole
|
||||
// block (and the benchmarks file) if you have no measured numbers yet.
|
||||
benchmarkCommands: {
|
||||
speed:
|
||||
`python3 -m sglang.bench_serving \\
|
||||
--backend sglang \\
|
||||
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||
--model {{MODEL_NAME}} \\
|
||||
--dataset-name {{DATASET}} \\
|
||||
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
|
||||
// One entry per accuracy field. A value is a string, OR a {[variant]: string}
|
||||
// object when the command differs per variant. Keys must match ACCURACY_LABELS
|
||||
// in _deployment.jsx + the per-cell/defaultAccuracy keys.
|
||||
accuracy: {
|
||||
gsm8k_pct:
|
||||
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
|
||||
sgl-eval run gsm8k \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||
--num-threads 32`,
|
||||
},
|
||||
// {{NUM_PROMPTS}} fallback per concurrency (else max(c*2, 200)).
|
||||
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 },
|
||||
},
|
||||
|
||||
// OPTIONAL — per-variant accuracy applied to EVERY cell of a variant (hardware-
|
||||
// independent, e.g. GPQA/AIME). Per-cell `accuracy` overrides. Keys must match
|
||||
// ACCURACY_LABELS + benchmarkCommands.accuracy. Delete if no numbers yet.
|
||||
defaultAccuracy: {
|
||||
default: { gsm8k_pct: null },
|
||||
},
|
||||
|
||||
// OPTIONAL — `# ...` hint lines prepended to multi-node commands, ONLY for hw
|
||||
// whose fabric needs manual NIC env (e.g. gb200 NVL72/MNNVL). NOT every multi-N
|
||||
// hw needs this — standard-IB DeepEP / Marlin multi-node don't. Delete if unused.
|
||||
multiNodeHints: {
|
||||
gb200: [
|
||||
"The following env vars may be needed depending on your cluster:",
|
||||
" GLOO_SOCKET_IFNAME=<your-nic>",
|
||||
" NVSHMEM_ENABLE_NIC_PE_MAPPING=1",
|
||||
" NVSHMEM_HCA_LIST=<your-hca-list>",
|
||||
],
|
||||
},
|
||||
|
||||
// Per-hw image for `docker run` framing. ASK the user which sglang build the recipes ran
|
||||
// on; don't guess a supporting release. Default below is :dev (nightly) — replace the tag
|
||||
// with the user's release if they give one. NVIDIA share one image; AMD uses ROCm tags.
|
||||
// GB200/GB300/B300 may need a `-cu130` (CUDA 13) tag — confirm per release.
|
||||
dockerImages: {
|
||||
h100: "lmsysorg/sglang:dev",
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
b200: "lmsysorg/sglang:dev",
|
||||
b300: "lmsysorg/sglang:dev",
|
||||
gb200: "lmsysorg/sglang:dev",
|
||||
gb300: "lmsysorg/sglang:dev",
|
||||
mi300x: "lmsysorg/sglang:dev-rocm720-mi30x",
|
||||
mi325x: "lmsysorg/sglang:dev-rocm720-mi30x",
|
||||
mi350x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
||||
mi355x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
||||
},
|
||||
|
||||
// Pre-selects the issue template's `model` dropdown on "Submit verified cell".
|
||||
// Must match that dropdown's value (usually `<hf-org>/<model-slug>`).
|
||||
github: {
|
||||
cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
|
||||
},
|
||||
|
||||
// Opt-in per axis. DELETE any axis your model doesn't expose (don't leave a stub).
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
null, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
{ id: "cp", label: "CP", values: [null, 1, 2, 4] },
|
||||
{ id: "dpAttn", label: "DP-Attention",
|
||||
values: [
|
||||
null, false, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
],
|
||||
labels: { "auto": "Auto", "false": "Off" } },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "MoE Parallelism" ----- KEEP if MoE + multiple `--moe-*-backend`
|
||||
// choices. DELETE for dense models.
|
||||
moe: {
|
||||
backend: {
|
||||
options: [
|
||||
{ id: null, label: "Inherited" },
|
||||
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
|
||||
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"] },
|
||||
{ id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] },
|
||||
],
|
||||
},
|
||||
ep: { label: "EP", values: [
|
||||
null, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
},
|
||||
|
||||
// ----- Card: "Parsers" ----- KEEP if the model has reasoning / tool-call
|
||||
// parsers (set the slugs below). DELETE the axis if neither applies.
|
||||
parsers: {
|
||||
items: [
|
||||
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser __REASONING_PARSER__" },
|
||||
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser __TOOLCALL_PARSER__" },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "Speculative Decoding" ----- KEEP if the model has spec-decoding
|
||||
// presets. Drop options the model doesn't support.
|
||||
speculative: {
|
||||
options: [
|
||||
{ id: "current", label: "Inherited from base" },
|
||||
{ id: "off", label: "Off (greedy)" },
|
||||
{ id: "mtp", label: "EAGLE / MTP",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
|
||||
{ id: "ngram", label: "NGRAM",
|
||||
flags: ["--speculative-algorithm NGRAM",
|
||||
"--speculative-num-draft-tokens 16",
|
||||
"--speculative-ngram-max-bfs-breadth 10"],
|
||||
disable: { dpAttnOn: [true] },
|
||||
disableReason: "NGRAM is incompatible with DP-Attention. Turn DP-Attention off in the Attention card above to use NGRAM." },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "PD Disaggregation" ----- KEEP if the model supports prefill/
|
||||
// decode disaggregation. Delete `router` if you have no router topology.
|
||||
pdDisagg: {
|
||||
modes: [
|
||||
{ id: "off", label: "Off" },
|
||||
{ id: "prefill", label: "Prefill role" },
|
||||
{ id: "decode", label: "Decode role" },
|
||||
],
|
||||
transferBackends: [
|
||||
{ id: "mooncake", label: "Mooncake",
|
||||
env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"],
|
||||
envWhen: { hw: ["gb200", "gb300"] } },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
// `auto` is a sentinel (emits no --disaggregation-ib-device flag).
|
||||
ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
|
||||
// Router fronting prefill + decode; substitute <prefill-host>/<decode-host>.
|
||||
router: {
|
||||
port: 8000,
|
||||
command:
|
||||
`python3 -m sglang_router.launch_router \\
|
||||
--pd-disaggregation \\
|
||||
--prefill http://<prefill-host>:30000 \\
|
||||
--decode http://<decode-host>:30001 \\
|
||||
--host 0.0.0.0 --port 8000 \\
|
||||
--disable-circuit-breaker \\
|
||||
--health-check-interval-secs 999999`,
|
||||
},
|
||||
},
|
||||
|
||||
// ----- Card: "Hierarchical KV Cache" ----- KEEP if the model is large enough
|
||||
// that hierarchical KV caching matters.
|
||||
hicache: {
|
||||
backends: [
|
||||
{ id: null, label: "Auto" },
|
||||
{ id: "file", label: "File" },
|
||||
{ id: "mooncake", label: "Mooncake" },
|
||||
{ id: "hf3fs", label: "HF3FS" },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
writePolicies: [
|
||||
{ id: "auto", label: "Auto" },
|
||||
{ id: "write_through", label: "Write-through" },
|
||||
{ id: "write_back", label: "Write-back" },
|
||||
{ id: "write_through_selective", label: "Write-through (selective)" },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "HiSparse" ----- KEEP only for DSA-style sparse-attention models
|
||||
// (DeepSeek-V3.2/V4, GLM-5). Decode-only: shown when live PD-Disagg mode is `decode`.
|
||||
hisparse: {
|
||||
requiredFlags: ["--disable-radix-cache"],
|
||||
config: { top_k: 2048, device_buffer_size: 6144 },
|
||||
hostRatios: [
|
||||
{ id: 5, label: "5 (~1TB host)" },
|
||||
{ id: 10, label: "10 (~2TB host)" },
|
||||
],
|
||||
defaultHostRatio: 10,
|
||||
},
|
||||
|
||||
// ----- Card: "MegaMoE" ----- KEEP only for Blackwell MoE kernel-fusion models.
|
||||
megamoe: {
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"],
|
||||
excludesStrategy: ["low-latency", "balanced"],
|
||||
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
|
||||
options: [
|
||||
{ id: "disabled", label: "Disabled" },
|
||||
{ id: "w4a8", label: "W4A8",
|
||||
flags: ["--moe-a2a-backend megamoe"],
|
||||
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
// EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
|
||||
// your model's verified recipe, or DELETE families you don't support. `match`
|
||||
// MUST have exactly the 5 keys; env/flags are flat literals.
|
||||
cells: [
|
||||
// ==== NVIDIA Blackwell + FP4 (single node) ====
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: true, // EXAMPLE — set false / replace with your verified recipe
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--moe-runner-backend flashinfer_mxfp4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== NVIDIA Hopper + FP8 (single node, DP-attention + DeepEP) ====
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== NVIDIA Hopper + FP4 (single node, Marlin W4A16 — Hopper has no FP4 runner) ====
|
||||
{
|
||||
match: { hw: "h100", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--moe-runner-backend marlin",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== AMD + BF16 (single node) — Triton attention + AITER; EP == TP for MoE ====
|
||||
{
|
||||
match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: ["SGLANG_USE_AITER=1", "SGLANG_ROCM_FUSED_DECODE_MLA=0"],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--ep 8",
|
||||
"--attention-backend triton",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== Multi-node example (2 nodes, TP=16) — engine injects --nnodes/--node-rank/
|
||||
// --dist-init-addr from match.nodes; do NOT add them here. ====
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" },
|
||||
verified: true, // EXAMPLE
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: __MODEL_DISPLAY__
|
||||
description: "__ONE_LINER__"
|
||||
tag: NEW
|
||||
mode: wide
|
||||
---
|
||||
|
||||
{/* TEMPLATE — instantiate via the cookbook-add-model skill, then DELETE this banner.
|
||||
(Frontmatter MUST stay the first thing in the file, so this note lives below it.)
|
||||
Replace every __TOKEN__, fill the TODO prose, delete the §3 subsections your model
|
||||
lacks. Tokens: __MODEL_DISPLAY__ __ONE_LINER__ __HF_ORG__ __MODEL_SLUG__ __HF_REPO__
|
||||
__REASONING_PARSER__ __TOOLCALL_PARSER__. MDX rules (JSX tables, labeled fences, no
|
||||
Docusaurus/@site/GitHub-alert/pipe-tables):
|
||||
.claude/skills/cookbook-add-model/references/mintlify-authoring.md */}
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
{/* TODO: 1-2 paragraph intro from the HF card — what the model is, release date,
|
||||
license, architecture highlights, context length. Keep it lean. */}
|
||||
**__MODEL_DISPLAY__** is __ONE_LINER__.
|
||||
|
||||
{/* TODO: variants table (JSX, NOT a markdown pipe table). Drop the table if there's
|
||||
a single variant and inline the HF link in the intro paragraph above instead. */}
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/__HF_ORG__/__HF_REPO__">__MODEL_DISPLAY__</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>TODO</td>
|
||||
<td style={{padding: "9px 12px"}}>TODO</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** {/* TODO e.g. `temperature=1.0`, `top_p=1.0` (informational; do NOT hardcode in sample code) */}
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/__HF_ORG__/__HF_REPO__).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
{/* TODO: model/hardware-specific tuning notes, caveats, known issues. Delete if none. */}
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
{/* Keep only the subsections that apply. Each runnable block is followed by an
|
||||
**Output Example:** + a ```text Output block with REAL server output. */}
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer.
|
||||
|
||||
{/* This example assumes a SEPARATE-FIELD parser (thinking → `reasoning_content`,
|
||||
answer → `content`). If your parser emits inline `<think>...</think>` tags inside
|
||||
`content`, parse the tags from `content` instead. */}
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="__HF_ORG__/__HF_REPO__",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
TODO: paste real server output here.
|
||||
```
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
{/* TODO: tool-calling example + **Output Example:**. On thinking-mode models the
|
||||
follow-up may put text in `reasoning_content`; print both that and `content`. */}
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
{/* TODO: keep only if the model is large enough for hierarchical KV caching; link
|
||||
the HiCache card in the Playground. Otherwise delete this subsection. */}
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
name: cookbook-review-pr
|
||||
description: Review a pull request against the SGLang Cookbook (docs_new/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
|
||||
---
|
||||
|
||||
# Cookbook Review PR
|
||||
|
||||
Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
|
||||
shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
|
||||
model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
|
||||
plus an MDX page. This checklist targets that layout. Field-schema detail lives in
|
||||
`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
|
||||
than restating.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
|
||||
2. `gh pr diff <N> --repo sgl-project/sglang`
|
||||
3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
|
||||
4. Run every checklist item against the diff.
|
||||
5. Output per-file verdicts + overall recommendation.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. File hygiene
|
||||
- A cookbook PR should only touch: `docs_new/src/snippets/configs/<vendor>/*.jsx`
|
||||
(config + benchmarks), `docs_new/cookbook/**/*.mdx`, `docs_new/docs.json`,
|
||||
`docs_new/cookbook/<category>/intro.mdx` (vendor card), `docs_new/cards/logos/<vendor>.png`
|
||||
(new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
|
||||
- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
|
||||
for unrelated commits accidentally included.
|
||||
- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
|
||||
model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
|
||||
(see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
|
||||
|
||||
### 2. Config quality (the per-model `config`)
|
||||
- Single `export const config = { ... }` literal — **no** function calls, spreads,
|
||||
fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
|
||||
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
- `github.cookbookModel` matches the issue-template `model` dropdown value.
|
||||
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
|
||||
(`megamoe` only on Blackwell MoE; `hisparse` only DSA-style; `pdDisagg.router` only with
|
||||
a PD topology).
|
||||
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||
nothing).
|
||||
- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
|
||||
greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
|
||||
|
||||
### 3. Cells / 5-dim matrix
|
||||
- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
|
||||
- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
|
||||
`commonFlags` reference (Mintlify won't inline it).
|
||||
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||
(the renderer injects them from `match.nodes`).
|
||||
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||
- Flag order: `--model-path` first, then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
last (the playground's insert anchors assume this).
|
||||
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||
|
||||
### 4. Benchmarks
|
||||
- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
|
||||
- `defaultAccuracy` keys ∈ `ACCURACY_LABELS` (and `benchmarkCommands.accuracy`).
|
||||
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||
that only released FP8/FP4 is a factual bug.
|
||||
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||
from the `sglang serve` deploy command.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
|
||||
### 5. Doc ↔ config parity (the #1 finding)
|
||||
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||
equal what the engine emits from the corresponding cell — same flags, same order. Drift
|
||||
here is the most common review miss.
|
||||
|
||||
### 6. Commands / port
|
||||
- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
|
||||
`python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
|
||||
guard against prose/cells reintroducing the old launcher.
|
||||
- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
|
||||
Launch port must match client/curl port on the same page.
|
||||
|
||||
### 7. Frontmatter
|
||||
- Every new MDX page has `title:` and `metatags.description:` (a real one-line value prop,
|
||||
not copied from another vendor).
|
||||
- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
|
||||
pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs_new/cookbook/`).
|
||||
- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
|
||||
- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
|
||||
"↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
|
||||
|
||||
### 8. Navigation & homepage
|
||||
- New page → `docs_new/docs.json` updated: under the right vendor group inside
|
||||
`navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
|
||||
`cookbook/<category>/<Vendor>/<Model>`.
|
||||
- Homepage `<Card href>` in `docs_new/cookbook/<category>/intro.mdx` points to the vendor's
|
||||
flagship; new vendors get a new `<Card>` + a logo at `docs_new/cards/logos/<vendor>.png` —
|
||||
**940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
|
||||
`git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
|
||||
- Don't change `docs_new/cookbook/intro.mdx` for individual model adds (top-level only).
|
||||
|
||||
### 9. Links & factual
|
||||
- HuggingFace URLs resolve to a real model. License section matches the actual HF license
|
||||
(don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
|
||||
The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
|
||||
guessed version.
|
||||
- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
|
||||
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||
not `${VAR}` (a bash no-op).
|
||||
|
||||
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||
blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
|
||||
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||
- Every runnable invocation block is followed by `**Output Example:**` + a `text Output`
|
||||
fenced block (real output, or `Pending update...` only with user acknowledgement).
|
||||
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||
|
||||
### 10. Quantization rules
|
||||
- FP4 is Blackwell-only (B200/B300/GB300) — never AMD; AMD FP4 chips must be `disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. FP8 configs adding `--kv-cache-dtype fp8_e4m3` should
|
||||
note the accuracy trade-off.
|
||||
|
||||
### 11. Scope
|
||||
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||
(e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
|
||||
the PR body.
|
||||
|
||||
### 12. Duplicate PRs
|
||||
- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
|
||||
risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
|
||||
|
||||
### 13. Build / validate
|
||||
```bash
|
||||
cd docs_new
|
||||
mint validate
|
||||
mint broken-links
|
||||
```
|
||||
Optional: `mint dev` for a visual smoke test.
|
||||
|
||||
### 14. Reviewer feedback
|
||||
- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
|
||||
addressed? Unresolved requested-changes should be flagged.
|
||||
|
||||
### 15. Grammar & spelling
|
||||
- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
|
||||
Flag each with the exact wrong text + correction.
|
||||
|
||||
## Output
|
||||
|
||||
Per file:
|
||||
- ✅ PASS
|
||||
- ⚠️ ISSUE: \<what\>
|
||||
- 🔴 BLOCK: \<what\>
|
||||
|
||||
Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**
|
||||
Reference in New Issue
Block a user