Cookbook renovation (#26885)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
6365d6faee
commit
d1777d1f6d
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: cookbook-add-model
|
||||
description: Add a new model to the SGLang Cookbook (docs_new/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Add a model to the SGLang Cookbook
|
||||
|
||||
The cookbook is **config-driven**: two shared engines contain NO model-specific code —
|
||||
`docs_new/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
|
||||
`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
|
||||
a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
|
||||
that imports them. No engine edits.
|
||||
|
||||
**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
|
||||
template is decoupled and covers all hardware + all axes):
|
||||
- `templates/config.jsx.tmpl` → `docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx`
|
||||
- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
|
||||
- `templates/page.mdx.tmpl` → `docs_new/cookbook/<category>/<Vendor>/<ModelName>.mdx`
|
||||
|
||||
The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
|
||||
lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
|
||||
*instance* you can consult, but is not the template.
|
||||
|
||||
**Deep references (read on demand, don't inline):**
|
||||
- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
|
||||
- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
|
||||
- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
|
||||
- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ cookbook/<category>/<Vendor>/<Model>.mdx │
|
||||
│ import { Deployment } from "/src/snippets/_deployment.jsx"; │
|
||||
│ import { Playground } from "/src/snippets/_playground.jsx"; │
|
||||
│ import { config } from "/src/snippets/configs/.../X.jsx"; │
|
||||
│ <Deployment config={config} /> <Playground config={config} />│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (config passed as React prop)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ src/snippets/configs/<vendor>/<model>.jsx │
|
||||
│ export const config = { │
|
||||
│ supportedHardware, variants, quantizations, strategies, ... │
|
||||
│ cells: [ { match:{hw,variant,quant,strategy,nodes}, │
|
||||
│ env:[...], flags:[...] }, ... ], // 5-dim matrix │
|
||||
│ playgroundFeatures: { attention, moe, parsers, ... }, │
|
||||
│ }; │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ (consumed by BOTH engines — no model code in engines)
|
||||
▼
|
||||
┌──────────────────────────────────┬──────────────────────────────┐
|
||||
│ _deployment.jsx │ _playground.jsx │
|
||||
│ Renders the verified matrix; │ Renders override chips + │
|
||||
│ one cell → its env/flags. │ diff against the cell. │
|
||||
└──────────────────────────────────┴──────────────────────────────┘
|
||||
```
|
||||
|
||||
The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
|
||||
playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
|
||||
every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
|
||||
shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
|
||||
|
||||
> The template is **autoregressive**. Diffusion / omni pages follow their own category
|
||||
> structure — don't force the config-driven template on them; still obey the Mintlify /
|
||||
> NEW-tag / docs.json / category-card / validation rules below.
|
||||
|
||||
---
|
||||
|
||||
**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
|
||||
everything upfront.** The real work is the verified `cells[]` recipes + measured
|
||||
benchmarks (Phases 2 + 4); everything else is filling the template.
|
||||
|
||||
## Phase 1 — Collect inputs
|
||||
|
||||
1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
|
||||
param count, architecture, context length, **license**. (Fetching guards factual bugs
|
||||
like an off-by-a-few-B param count.) If the model isn't public, ask the user.
|
||||
2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
|
||||
Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
|
||||
Default to BF16 when a full-precision repo exists.
|
||||
3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
|
||||
for each. List only tested hw (unlisted greys out).
|
||||
4. **Verified launch recipes** — the full `sglang serve` flags per
|
||||
(hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
|
||||
`python -m sglang.launch_server` to `sglang serve` form.
|
||||
5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
|
||||
(a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
|
||||
tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
|
||||
default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
|
||||
6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
|
||||
|
||||
**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
|
||||
this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
|
||||
(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
|
||||
|
||||
| Platform | Vendor | VRAM | Docker image |
|
||||
|---|---|---|---|
|
||||
| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
|
||||
| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
|
||||
| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
|
||||
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
|
||||
- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
|
||||
default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
|
||||
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
|
||||
hosts are typically **4 GPUs** (TP=4 ceiling).
|
||||
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||
- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
|
||||
`--ep` unless benchmarked. (The template's AMD example cell shows these.)
|
||||
|
||||
## Phase 2 — Instantiate the template
|
||||
|
||||
1. **Copy** the three template files to their target paths (above). Note the two
|
||||
vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
|
||||
2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
|
||||
`__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
|
||||
none remain: `grep -rn '__[A-Z_]*__' <new files>`.
|
||||
3. **Prune** to what the model supports (delete, don't stub) — using
|
||||
[references/authoring-reference.md](references/authoring-reference.md):
|
||||
- `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
|
||||
`mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
|
||||
(e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
|
||||
- `playgroundFeatures` axes: delete `megamoe` (non-Blackwell-MoE), `hisparse`
|
||||
(non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
|
||||
- `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
|
||||
to single `default` if there's no variant axis (then drop the `variant` half of
|
||||
`modelNames`/`defaultAccuracy` keys).
|
||||
4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
|
||||
set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
|
||||
`dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
|
||||
a guessed release), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
|
||||
|
||||
### Site-wiring (do all three)
|
||||
|
||||
- **`docs_new/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
|
||||
the **top** of that vendor's `pages` (root-relative, no `.mdx`:
|
||||
`cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
|
||||
local ordering.
|
||||
- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
|
||||
vendor dir for existing NEW and strip it from siblings; verify ≤1:
|
||||
`grep -rn 'tag: NEW' docs_new/cookbook/<category>/<Vendor>/` → at most one result. (Scan
|
||||
files; don't assume the first `docs.json` entry holds NEW.)
|
||||
- **Homepage card** — `docs_new/cookbook/<category>/intro.mdx`: if the org already has a
|
||||
`<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
|
||||
(title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
|
||||
ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
|
||||
transparent** PNG → `docs_new/cards/logos/<org-slug>.png` per
|
||||
[references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
|
||||
is gitignored repo-wide). Never invent or copy a logo.
|
||||
|
||||
## Phase 3 — Validate
|
||||
|
||||
```bash
|
||||
cd docs_new
|
||||
mint validate # frontmatter, missing nav entries, MDX/JSX errors
|
||||
mint broken-links
|
||||
mint dev # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
|
||||
```
|
||||
|
||||
Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
|
||||
Playground inherits the Deploy selection live; each axis toggle produces the expected
|
||||
diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
|
||||
hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
|
||||
and is gone from same-vendor siblings; the homepage card points to the new model.
|
||||
|
||||
## Phase 4 — Interactive testing
|
||||
|
||||
The user deploys each cell, runs the benches, and pastes results; you fill the data:
|
||||
- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
|
||||
- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
|
||||
speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
|
||||
`0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
|
||||
cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
|
||||
|
||||
## Phase 5 — Prose & config tips
|
||||
|
||||
**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
|
||||
carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
|
||||
rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
|
||||
user notes: §1 Model Introduction (description, links, params, license, variants table),
|
||||
§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
|
||||
Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
|
||||
parser's output shape; each runnable block gets an `**Output Example:**`).
|
||||
|
||||
## Phase 6 — Review
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Git workflow
|
||||
|
||||
Always branch — never commit to main directly.
|
||||
|
||||
```bash
|
||||
git checkout -b add-<model>-cookbook
|
||||
git add docs_new/src/snippets/configs/<hf-org>/<slug>.jsx \
|
||||
docs_new/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
|
||||
docs_new/cookbook/<category>/<Vendor>/<Model>.mdx \
|
||||
docs_new/docs.json docs_new/cookbook/<category>/intro.mdx
|
||||
git commit -m "Add <Display-Name> cookbook"
|
||||
git push -u origin add-<model>-cookbook
|
||||
gh pr create --title "Add <Display-Name> cookbook" --body "..."
|
||||
```
|
||||
@@ -0,0 +1,220 @@
|
||||
# Cookbook config reference (fields · cells · playground · MDX)
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill. This is the field-by-field
|
||||
contract for when the clone needs more than a rename. The two engine files are
|
||||
the canonical specs — read their headers first:
|
||||
|
||||
- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the 5-dim matrix widget; lists every config field.
|
||||
- [`_playground.jsx`](../../../../docs_new/src/snippets/_playground.jsx) — the diff-based override widget; lists the `playgroundFeatures` axes + the `AXIS_HANDLERS` interface.
|
||||
|
||||
Engine extension (adding a new playground axis) lives in [engine-axis.md](engine-axis.md).
|
||||
|
||||
---
|
||||
|
||||
## 2.1 Create the config file
|
||||
|
||||
**Path**: `docs_new/src/snippets/configs/<vendor>/<model>.jsx`. The vendor folder is
|
||||
the HuggingFace org (`deepseek-ai`, `Qwen`, `moonshotai`, ...); the file
|
||||
name is a short hyphenated model id (`deepseek-v4`, `qwen3.5`, ...).
|
||||
|
||||
**Shape**: must be a single `export const config = { ... }` literal. Do not
|
||||
use function calls, spreads, fragment refs, or IIFE — Mintlify re-evaluates
|
||||
this export at hydration time with module-level identifiers out of scope,
|
||||
and any non-literal value crashes with `ReferenceError`.
|
||||
|
||||
**Required fields** (engine reads these — see the `_deployment.jsx` header for
|
||||
the full contract):
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
|
||||
| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
|
||||
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
|
||||
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
|
||||
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Common ids: `low-latency`, `balanced`, `high-throughput`. |
|
||||
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
|
||||
| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. |
|
||||
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
|
||||
| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
|
||||
| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
|
||||
|
||||
**Optional fields**:
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
|
||||
| `dockerImages` | `{[hwId]: string}` | Per-hw image name for `docker run` framing. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `ACCURACY_LABELS` + `benchmarkCommands.accuracy`. |
|
||||
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the value that matches the `model` dropdown in your issue template so it's pre-selected when the issue opens. |
|
||||
|
||||
## 2.2 Author the 5-dim matrix (`cells[]`)
|
||||
|
||||
Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
|
||||
```js
|
||||
{
|
||||
match: { hw: "b200", variant: "flash", quant: "fp4",
|
||||
strategy: "low-latency", nodes: "single" },
|
||||
verified: true, // green "Verified" badge; absence = yellow
|
||||
env: [
|
||||
"SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024",
|
||||
],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}", // {{MODEL_NAME}} resolves from modelNames
|
||||
"--tp 4",
|
||||
"--moe-runner-backend flashinfer_mxfp4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
|
||||
- `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`,
|
||||
`strategy`, `nodes`. The engine looks up cells by tuple equality.
|
||||
- `env` and `flags` are FLAT literals. The engine does NOT expand
|
||||
fragments, aliases, or templates — it consumes them verbatim
|
||||
(only `{{PLACEHOLDER}}` substitutions happen at render time).
|
||||
- DO NOT include `--nnodes` / `--node-rank` / `--dist-init-addr` in
|
||||
`cell.flags` for multi-node cells. The renderer injects them
|
||||
automatically from `match.nodes` (`multi-N` → N nodes).
|
||||
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
|
||||
`{{PORT}}` placeholders so users can override through the Env modal.
|
||||
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
|
||||
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
|
||||
flags, then tuning knobs, with `--host` / `--port` last. The playground
|
||||
engine assumes this ordering when inserting overrides (its anchors target
|
||||
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
|
||||
|
||||
**Cells are denormalized on purpose** — common flags repeat across cells.
|
||||
This makes each cell self-contained and easy to verify. When sweeping a
|
||||
common change, edit every cell.
|
||||
|
||||
**Avoid premature cells**: only add a cell for a (hw × variant × quant ×
|
||||
strategy × nodes) combination if you have a recipe that has been tested or
|
||||
at least sanity-checked. The engine greys out un-listed combinations
|
||||
automatically.
|
||||
|
||||
## 2.3 Configure `playgroundFeatures` (optional)
|
||||
|
||||
The Playground widget is opt-in per axis. Add only the axes that make sense
|
||||
for this model. Recognised axis keys and their schemas (full reference in
|
||||
the `_playground.jsx` header):
|
||||
|
||||
| Axis key | Widget | Use when |
|
||||
|---|---|---|
|
||||
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
|
||||
| `moe` | Backend select + EP knob | Model is MoE and supports multiple `--moe-*-backend` choices. |
|
||||
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
|
||||
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
|
||||
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
|
||||
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
|
||||
| `megamoe` | Single-select with hw/strategy gating | Blackwell-only kernel fusion variant. |
|
||||
|
||||
**Per-chip constraints**: any chip entry in any axis can be wrapped with
|
||||
`hide` / `disable` constraint objects:
|
||||
|
||||
```js
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }
|
||||
```
|
||||
|
||||
- `hide` — chip omitted entirely (use for hard impossibilities).
|
||||
- `disable` — chip greyed out with tooltip (soft warning).
|
||||
- Constraints are AND across keys, OR within each key's array.
|
||||
- Bare `disabled: true` / `disable: true` is a static always-disabled form
|
||||
(used for "Coming soon" chips).
|
||||
|
||||
## 2.4 Create the MDX page
|
||||
|
||||
Path: `docs_new/cookbook/<category>/<Vendor>/<Model>.mdx`. Import both widgets and
|
||||
the per-model config, render them inside the relevant sections:
|
||||
|
||||
```mdx
|
||||
## Deployment
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/<vendor>/<model>.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/<vendor>/<model>-benchmarks.jsx";
|
||||
|
||||
{/* Install is a PREREQUISITE — keep it compact + collapsed at the top of the
|
||||
Deploy section (NOT a numbered section). Tabs mirror the widget's
|
||||
Python/Docker toggle. */}
|
||||
<a id="install" />
|
||||
<Accordion title="Install SGLang">
|
||||
<Tabs>
|
||||
<Tab title="Python (pip / uv)">…pip / uv install…</Tab>
|
||||
<Tab title="Docker">…docker pull + a `docker run … sglang serve` example…</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
[model-specific tuning notes, caveats, links]
|
||||
|
||||
## Playground
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
```
|
||||
|
||||
**Heading slugs matter** — the two widgets cross-link by scrolling to each
|
||||
other's section id (Mintlify auto-slugs headings: lowercase, spaces →
|
||||
hyphens, punctuation dropped). The engines look up:
|
||||
|
||||
- the **Deploy** panel by id `deployment` (falls back to `deploy`) — used
|
||||
by the Playground's "↑ Switch base" button and by deep-link scroll-on-
|
||||
load. Title the section `## Deployment` (or `## Deploy`).
|
||||
- the **Playground** by id `playground` — used by `_deployment.jsx`'s
|
||||
"Open the Playground →" link. Title the section `## Playground`.
|
||||
|
||||
Avoid numbered headings like `## 3. Model Deployment` (slug
|
||||
`3-model-deployment`) for these two sections — the cross-links would break.
|
||||
The Playground reads the Deploy selection live via the URL hash + the
|
||||
`sglang-deploy-sel` custom event, so the two can live in different parts
|
||||
of the page.
|
||||
|
||||
The `benchmarks` prop is **optional**. It points at a sibling
|
||||
`<model>-benchmarks.jsx` file (one entry per cell, keyed by the same
|
||||
`match` tuple) that renders an accuracy + speed sub-card under the command
|
||||
box; omit the import and the prop if the cookbook has no measured numbers
|
||||
yet. See the `_deployment.jsx` header and `deepseek-v4-benchmarks.jsx` for
|
||||
the full speed/accuracy schema.
|
||||
|
||||
To let users *reproduce* those numbers, add a `benchmarkCommands` block to
|
||||
the config (§2.1, next to `curl`). When present alongside `benchmarks`, the
|
||||
benchmark card grows a **"⚡ Reproduce"** button that opens a modal listing
|
||||
the runnable commands for the current cell — one `bench_serving` command for
|
||||
Speed (with concurrency chips that rewrite `--max-concurrency`) plus an
|
||||
Accuracy command with a chip per eval. No separate benchmark section needed.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls (authoring)
|
||||
|
||||
**Stale URL hash hydration** — If a user shares a link from an old cell
|
||||
catalog and the hash names an impossible combination, `_deployment.jsx`'s
|
||||
`validateSelection` snaps to the nearest real cell. The Playground reads
|
||||
the hash too — make sure cookbook removals don't leave dangling shared
|
||||
links pointing at hardware/quant combos that no longer exist.
|
||||
|
||||
**Mintlify constraints** — Module-level statements are stripped. The config
|
||||
MUST be a single `export const config = { ... }` literal — no function calls,
|
||||
spreads, fragment refs, or IIFE (Mintlify re-evaluates the export at hydration
|
||||
with module-level identifiers out of scope; any non-literal crashes with
|
||||
`ReferenceError`). In MDX, capitalized JSX tags get rebound — use the built-in
|
||||
Mintlify components (`<Accordion>`, `<Tabs>`, `<Card>`, ...) as documented.
|
||||
Avoid `!(x in y)` anywhere (Mintlify's AST walker crashes on it) — use
|
||||
`obj.key === undefined`.
|
||||
|
||||
**Per-cell denormalization** — Cells repeat common flags on purpose. Do
|
||||
not factor them into a shared `commonFlags` array — Mintlify will fail
|
||||
to inline the reference. If you need to sweep a flag across cells, do it
|
||||
with a global find-replace in the config file.
|
||||
@@ -0,0 +1,242 @@
|
||||
# Engine extension: add a new playground feature axis
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill. **Rare** — adding a model
|
||||
cookbook is data-only and never needs this. The current 8 built-in axes
|
||||
(`attention`, `moe`, `parsers`, `speculative`, `pdDisagg`, `hicache`,
|
||||
`hisparse`, `megamoe`) already cover the SGLang feature surface most cookbooks
|
||||
need. Only add a new axis if a real cookbook needs it and the feature does not
|
||||
fit any existing axis. Touches `_playground.jsx` only.
|
||||
|
||||
For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
|
||||
|
||||
---
|
||||
|
||||
## 3.1 Decide
|
||||
|
||||
Before touching the engine, confirm:
|
||||
|
||||
- The feature is a STABLE part of the SGLang CLI surface (will appear in
|
||||
multiple cookbooks, not one-off).
|
||||
- The feature cannot be expressed as a new option inside an existing axis
|
||||
(e.g. a new MoE backend belongs in `moe.backend.options`, not a new
|
||||
axis).
|
||||
- The feature has a clean strip-prefix → emit-flag pattern.
|
||||
|
||||
If unsure, add it as data first (in one cookbook's config under an
|
||||
existing axis) before promoting it to a built-in axis.
|
||||
|
||||
## 3.2 Pick the axis id and state shape
|
||||
|
||||
The axis id is the key in both `config.playgroundFeatures` and the
|
||||
internal `deltas` object. Use camelCase, descriptive but short:
|
||||
`mambaCache`, `attentionBackend`, `kvCacheDtype`.
|
||||
|
||||
The state shape is whatever `initState` returns. Common shapes:
|
||||
|
||||
- Single-select: a string sentinel (e.g. `"disabled"` / `"current"` / an
|
||||
option id).
|
||||
- Multi-toggle: `{[itemId]: bool}`.
|
||||
- Sub-knobs: `{[knobId]: value | null}`.
|
||||
- Compound (axis with its own internal sub-state, like PD-Disagg's
|
||||
`{mode, ibDevice}`): a plain object.
|
||||
|
||||
Pick ONE "inherit base" sentinel and document it in the handler comment.
|
||||
|
||||
## 3.3 Implement the handler
|
||||
|
||||
Add one entry to `AXIS_HANDLERS` in `_playground.jsx`.
|
||||
The handler owns everything: state init, apply (strip+insert), hidden-revert,
|
||||
AND the JSX render. Engine main loop iterates `AXIS_HANDLERS` and calls each
|
||||
method by name — adding a new axis is genuinely a one-place change.
|
||||
|
||||
Template:
|
||||
|
||||
```js
|
||||
// ---- Axis: <Title> ----------------------------------------------------
|
||||
// <one-paragraph description of what this axis controls and why it
|
||||
// exists. Mention the SGLang feature it wraps and the strip/insert
|
||||
// policy.>
|
||||
<axisId>: {
|
||||
initState: (fc) => /* initial state value */,
|
||||
|
||||
// Called when base cell changes. Return new value if the picked option
|
||||
// is now hidden by a constraint; otherwise return value unchanged.
|
||||
// Disabled picks are intentionally NOT auto-reverted (soft warning).
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
// ... return value or a new value
|
||||
return value;
|
||||
},
|
||||
|
||||
// Pure function. Receives the current (flags, env) and returns the next
|
||||
// (flags, env). Do NOT mutate inputs. The `value` argument is whatever
|
||||
// initState returned. The `fc` argument is config.playgroundFeatures[axisId].
|
||||
// The `sel` argument is the current base cell selection. The `h`
|
||||
// argument is the helpers bundle (strip/insert primitives + anchors).
|
||||
apply: ({ flags, env, value, fc, sel, h, derived }) => {
|
||||
if (/* value is the inherit-base sentinel */) return { flags, env };
|
||||
flags = h.stripFlagsByFirstToken(flags, [/* prefixes this axis owns */]);
|
||||
if (/* an option is picked */) {
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_<X>, [/* new flags */]);
|
||||
// or: flags = h.insertBeforeTail(flags, [/* new flags */]);
|
||||
// if the axis mutates env:
|
||||
// env = h.stripEnvByPrefix(env, fc.stripEnv || []);
|
||||
// env = [...env, /* additional env vars */];
|
||||
}
|
||||
return { flags, env };
|
||||
},
|
||||
|
||||
// Optional: read the base cell's flag array back into the same shape
|
||||
// initState/apply use. Render shows this as the default selection
|
||||
// (dropdown option or checked chip) when the state slot is the inherit
|
||||
// sentinel — so the user sees the cell's actual --tp / MoE backend /
|
||||
// spec preset instead of an opaque "Auto." When derive returns a real
|
||||
// value, the inherit-sentinel option is hidden from the control. Apply
|
||||
// also receives the derived
|
||||
// value (as `derived`) and may use it as a no-op shortcut when the
|
||||
// user's pick matches base. Skip when your axis owns flags that never
|
||||
// appear in base cells (PD-Disagg / HiCache / MegaMoE).
|
||||
// deriveFromBase: (cell, fc, h) => ({ ... }) | null,
|
||||
|
||||
// Optional: hints for the renderer. Currently only pdDisagg uses this
|
||||
// to report its role banner. Omit if not needed.
|
||||
// getRenderHints: (value, fc) => ({ pdMode: ... }) | null,
|
||||
|
||||
// Returns the axis card JSX. The outer div MUST have key={axisId} so
|
||||
// React can track it in the engine's map loop. Return null for
|
||||
// axis-level gating (e.g. MegaMoE on Hopper). Lay out as a single
|
||||
// compact horizontal row: title on the left, fields after.
|
||||
render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => {
|
||||
if (/* axis-level gating fails */) return null;
|
||||
return (
|
||||
<div key={axisId} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
<span style={s.axisTitle}>Axis Title</span>
|
||||
{/* For multi-option fields, use renderSelect(...) — the default.
|
||||
For on/off toggles or single-select chip groups, use
|
||||
renderChip instead (see "Control choice" in the conventions
|
||||
below). Read state from `value`; write via `setValue(next)`
|
||||
(replaces the whole axis slot). */}
|
||||
<span style={s.field}>
|
||||
<span style={s.fieldLabel}>Field</span>
|
||||
{renderSelect(value.slot, fc.entries, (v) =>
|
||||
setValue({ ...value, slot: v }), base)}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
},
|
||||
},
|
||||
```
|
||||
|
||||
**Important conventions**:
|
||||
|
||||
- Insert the entry in the position you want it rendered. `AXIS_HANDLERS`
|
||||
is iterated in insertion order for both render and apply.
|
||||
- Use `h.ANCHOR_NEAR_*` constants for insertion. Add a new anchor to the
|
||||
helpers bundle if your axis needs to land somewhere new in the flag
|
||||
block.
|
||||
- Use lowercase HTML JSX tags only. Capitalized tags get rebound by
|
||||
Mintlify.
|
||||
- Inside `render`, read state via `value` (the slice for this axis).
|
||||
Write state via `setValue(next)` (replaces the whole slice). For
|
||||
compound axes, do `setValue({ ...value, [k]: nextK })`.
|
||||
- Layout: one `s.compactRow` per axis card, `s.axisTitle` for the
|
||||
leading label, one `s.field` per (label + input) pair.
|
||||
- Control choice — `renderSelect` vs `renderChip`:
|
||||
- `renderSelect(current, entries, onPick, base, labelFor?, opts?)` is
|
||||
the **default** compact control (a `<select>` dropdown). It filters
|
||||
hidden chips and disables greyed-out ones internally — no per-chip
|
||||
`evaluateChip` loop needed in the render body. Most axes use it
|
||||
(attention, moe, pdDisagg, hisparse, hicache, megamoe). Pass
|
||||
`{ hideValues: [<sentinel>] }` when your `deriveFromBase` resolved to
|
||||
a real value, so the inherit-sentinel ("Auto" / "Inherited" /
|
||||
"current") doesn't clutter the dropdown.
|
||||
- `renderChip(label, current, value, onPick, { disabled?, disabledReason? })`
|
||||
renders a **button** instead of a dropdown row. Use it for a chip
|
||||
group when you want the options laid out as buttons. It serves two
|
||||
shapes:
|
||||
- **Multi-toggle** (Parsers) — one independent on/off chip per item;
|
||||
`current` is that item's effective bool, `value` is `true`, so the
|
||||
chip is "checked" when the item is on.
|
||||
- **Single-select** (Speculative) — a radio-style group; pass the
|
||||
group's effective value as `current` and each option's id as
|
||||
`value`, so exactly one chip is checked (`current === value`).
|
||||
Chip groups own their visibility/disable filtering: loop
|
||||
`h.evaluateChip(opt, base)` in the render body, skip `c.hidden`,
|
||||
filter the inherit-sentinel yourself when `deriveFromBase` resolved
|
||||
to a real value, and forward `c.disabled` / `c.disableReason` into
|
||||
`renderChip`'s opts (this is what surfaces a disabled chip's tooltip,
|
||||
e.g. a "Coming soon" entry).
|
||||
- Selected chips use the same terracotta (`#D45D44`) as the Deploy
|
||||
panel's selected button, so both widgets read as one
|
||||
visual system. Don't introduce a per-axis accent color.
|
||||
- Default-from-base: if your axis can be read out of base cells'
|
||||
flags, implement `deriveFromBase` and have your render show the
|
||||
derived value when state is the sentinel (e.g.
|
||||
`const eff = value.tp !== null ? value.tp : (derived && derived.tp)`).
|
||||
This is what makes a fresh playground load show the user's actual
|
||||
recipe instead of "auto." Flag-parsing helpers on `h`:
|
||||
`parseIntFlag`, `hasFlag`, `findFlagArg`.
|
||||
- **Avoid the `in` operator wrapped in unary** (`!(x in y)`). Mintlify's
|
||||
AST walker crashes on it (`TypeError: this[e] is not a function`). Use
|
||||
`obj.key === undefined` or `obj.id !== undefined` instead. Bare
|
||||
`if (key in obj)` (no surrounding `!`) is fine.
|
||||
|
||||
## 3.4 Document the per-cookbook schema
|
||||
|
||||
Edit the file header in `_playground.jsx` to add your new axis to the
|
||||
"Recognised keys" list, with a one-line description of its schema.
|
||||
Optionally add a paragraph below explaining its strip/insert policy.
|
||||
|
||||
Update the §2.3 axis table in [authoring-reference.md](authoring-reference.md) to list the new axis.
|
||||
|
||||
## 3.5 Migrate cookbooks that need it
|
||||
|
||||
For each cookbook that should expose this axis, add a
|
||||
`playgroundFeatures.<axisId>` entry to its config. Verify the chip group
|
||||
renders, options apply correctly, and the diff matches expectations.
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls (engine work)
|
||||
|
||||
**Insertion anchor misses** — `insertAfter` falls back to right-after
|
||||
`--model-path` if none of its anchor prefixes are present. If your axis
|
||||
emits flags that should land somewhere specific, include the most likely
|
||||
anchor prefixes in your call. Order doesn't matter (set semantics).
|
||||
|
||||
**Conditional strips** — Some axes strip ONLY when overridden
|
||||
(`attention.tp`, `moe.backend`, `speculative`, `megamoe`). Others strip
|
||||
UNCONDITIONALLY whenever declared (`parsers`, `pdDisagg`, `hicache`). The
|
||||
header comment in `AXIS_HANDLERS` documents which policy each axis uses;
|
||||
follow the same pattern when adding a new axis. If unsure, prefer
|
||||
conditional strip — it preserves base behavior when the user does not
|
||||
opt in.
|
||||
|
||||
**Closure of `AXIS_HANDLERS`** — Inside a handler method, you can
|
||||
reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (megamoe
|
||||
does this for `_gateOpen`). This works because `AXIS_HANDLERS` is in
|
||||
lexical scope. Do NOT use this for general logic — it tightly couples
|
||||
handlers. Reserve it for one handler's helpers shared between its own
|
||||
`render` and `revertHidden`.
|
||||
|
||||
---
|
||||
|
||||
## Review checklist for a new-axis PR
|
||||
|
||||
- [ ] `AXIS_HANDLERS` is the ONLY place that mentions the new axis id
|
||||
(apart from per-cookbook config). No `if (axisId === '<new>')`
|
||||
branches anywhere in the engine.
|
||||
- [ ] `initState` is deterministic and idempotent (does not depend on
|
||||
the base cell).
|
||||
- [ ] `apply` is pure — does not mutate inputs.
|
||||
- [ ] `revertHidden` returns the same reference when nothing changed
|
||||
(avoids unnecessary re-renders).
|
||||
- [ ] `render` returns `null` when axis-level gating fails (whole card
|
||||
hidden) — does not render an empty placeholder.
|
||||
- [ ] `render` sets `key={axisId}` on its outer element.
|
||||
- [ ] No `!(x in y)` patterns introduced (Mintlify AST walker crashes).
|
||||
- [ ] File header lists the new axis in "Recognised keys".
|
||||
- [ ] The §2.3 table in `authoring-reference.md` lists the new axis.
|
||||
- [ ] One existing cookbook config is updated to consume the new axis,
|
||||
and visual verification shows the diff is correct.
|
||||
@@ -0,0 +1,107 @@
|
||||
# MDX authoring rules (Mintlify) + invocation-example patterns
|
||||
|
||||
Loaded on demand by the `cookbook-add-model` skill (Phase 5, writing the page prose).
|
||||
These are model-agnostic Mintlify hygiene rules — the most common review findings.
|
||||
The cookbook is **Mintlify**, not Docusaurus.
|
||||
|
||||
## Mintlify syntax
|
||||
|
||||
**Allowed components**: `<Card>`, `<CardGroup>`, `<Note>`, `<Tip>`, `<Warning>`,
|
||||
`<Info>`, `<Accordion>`, `<AccordionGroup>`, `<Steps>`, `<Step>`, `<Tabs>`, `<Tab>`,
|
||||
`<CodeGroup>`, `<Frame>`, `<Icon>`.
|
||||
|
||||
**Forbidden** (flag every occurrence):
|
||||
- Docusaurus admonitions (`:::note` / `:::warning` / …) — use `<Note>` / `<Warning>`.
|
||||
- `@site/...` / `@theme/...` imports — use absolute `/src/snippets/...`.
|
||||
- GitHub alert blocks (`> [!NOTE]`, `> [!WARNING]`).
|
||||
- **Markdown pipe tables** on new pages — use JSX `<table>` (see below).
|
||||
- Inline `<details>` / `<summary>` — use `<Accordion>`.
|
||||
- Unknown / non-Mintlify components.
|
||||
- `<CardGroup>` / `<Card>` on individual model pages — those are for category `intro.mdx` only.
|
||||
|
||||
**Code fences**: always labeled — ` ```python Example `, ` ```bash Command `,
|
||||
` ```shell Command `, ` ```text Output `. When nesting a fenced block inside another,
|
||||
the **outer** fence uses four backticks.
|
||||
|
||||
**Internal links**: root-relative, no extension (`/cookbook/<category>/<Vendor>/<Model>`);
|
||||
`docs.sglang.io` is canonical. Flag `.md`/`.mdx` extensions and `../`-relative page links
|
||||
in body prose. (Existing cookbook pages do use `../../../docs/...` for cross-links into
|
||||
the non-cookbook docs tree — that's the established exception; don't introduce new ones.)
|
||||
|
||||
## JSX tables (required for all tables on new pages)
|
||||
|
||||
```jsx
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Col</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "9px 12px"}}>cell</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
```
|
||||
|
||||
Alternate column background colors (`rgba(255,255,255,0.02)` / `0.05`) for readability;
|
||||
adjust `<colgroup>` widths for 3- or 5-column tables. The DeepSeek-V4 page §1 variants
|
||||
table is a live reference.
|
||||
|
||||
## Invocation-example patterns (§3 Advanced Usage)
|
||||
|
||||
- **Reasoning-parser output shape must match the example**:
|
||||
- *Separate-field* parsers (most qwen/glm, `kimi_k2`, `deepseek-v4`): thinking lands
|
||||
in `message.reasoning_content`, answer in `message.content` — print both.
|
||||
- *Inline-tag* parsers (e.g. `minimax-append-think`): thinking is wrapped in
|
||||
`<think>...</think>` **inside** `message.content` — the client parses the tags; for
|
||||
streaming, buffer and split on the markers.
|
||||
Pick the pattern from the model card / SGLang docs for that specific parser.
|
||||
- **Hybrid reasoning models**: show both thinking-on (default) and thinking-off
|
||||
(`extra_body={"chat_template_kwargs": {"thinking": False}}` or `enable_thinking: False`).
|
||||
- **Tool-call follow-up on thinking models**: the final assistant turn may put text in
|
||||
`reasoning_content` instead of (or with) `content` — print both so the output isn't a
|
||||
misleading `None`.
|
||||
- **Every runnable block** is immediately followed by `**Output Example:**` + a
|
||||
` ```text Output ` block with **real** server output (verbatim, not paraphrased).
|
||||
`Pending update...` is acceptable only with the user's explicit acknowledgement.
|
||||
- **Do not hardcode sampling params** (`temperature`, `top_p`) in sample code — SGLang
|
||||
uses `generation_config.json` defaults. Listing "Recommended generation" in §1 is fine.
|
||||
- Format raw API objects (`ChatCompletionMessage(...)`) into readable Reasoning /
|
||||
Content / Tool Calls blocks.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
- **Top-level `description:`** is the canonical field — it sets the page's SEO meta
|
||||
description (`og:`/`twitter:description` fall back to it) AND renders as the visible
|
||||
**subtitle** under the title, filling the header band before the first heading. Give every
|
||||
page a one-line top-level `description` (a lede / value prop) — without it, a page that
|
||||
opens straight into `## Deployment` shows an empty gap under the title (the title and
|
||||
`## Deployment` are the same size, so they read as two bare headings). Do **not** put the
|
||||
description inside a `metatags` block — `metatags` is for other/custom tags, and
|
||||
`metatags.description` is redundant with (and non-canonical vs) the top-level field.
|
||||
- **Write it for SEO** (it doubles as the search-result snippet): front-load the exact
|
||||
model name + intent — e.g. `Deploy <Model> with SGLang — …` — aim for ~150–160 chars, and
|
||||
pack secondary keywords (variants + sizes, `Mixture-of-Experts` / architecture, target
|
||||
GPUs). Phrase it as a value prop, not a generic "`<Model>` is a … model" intro.
|
||||
- Frontmatter MUST be the first thing in the file — no comment or blank line before the
|
||||
opening `---`.
|
||||
|
||||
## Commands & ports
|
||||
|
||||
- **Deploy/launch** commands use `sglang serve --model-path …` — never
|
||||
`python -m sglang.launch_server` / `python3 -m sglang.launch_server` (deprecated).
|
||||
- **Benchmark workload** commands use `python3 -m sglang.bench_serving …` (never bare
|
||||
`python -m`); built-in accuracy scripts use `python3 benchmark/...`.
|
||||
- Port **30000** everywhere on a page — launch, curl, client `base_url`, and bench must
|
||||
agree. Keep one canonical deploy command (the Deploy widget) and don't re-paste launch
|
||||
commands across sections; the documented command must match the widget's output for the
|
||||
same selection (doc ↔ config parity).
|
||||
|
||||
## Factual hygiene
|
||||
|
||||
- License must match the actual HuggingFace license (don't copy from another model).
|
||||
- HF URLs resolve to a real model; Docker images from `lmsysorg/sglang`.
|
||||
- No Google-Drive image links (they don't render); host images in the repo.
|
||||
- Shell placeholders are `export VAR=<value>`, not `export VAR=${VAR}` (a bash no-op).
|
||||
- `tag: NEW` is sparing — at most one per `<category>/<Vendor>/` dir (the newest); strip
|
||||
it from siblings when adding a new NEW page.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Vendor card logo (new brand only)
|
||||
|
||||
A new vendor/brand in the cookbook landing grid needs a card logo at
|
||||
`docs_new/cards/logos/<org-slug>.png`. **Ask the user for the brand's logo, then generate
|
||||
the conforming PNG** — never invent, copy, or hallucinate one, and never ship a
|
||||
non-conforming file. Reference: PR #27400 (added `tencent.png` + `poolside.png`).
|
||||
|
||||
If the org already has a card/logo, do nothing here — only update the `<Card href>`.
|
||||
|
||||
## Spec (match the existing logos exactly)
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Path | `docs_new/cards/logos/<org-slug>.png` — lowercase, matches the `img=` in the `<Card>` |
|
||||
| Canvas | **940 × 525** px |
|
||||
| Mode | **RGBA**, fully **transparent** background |
|
||||
| Content | **Icon-only** — the brand glyph/mark (the "swirl"), **no wordmark text** |
|
||||
| Placement | centered; glyph ≈ 0.33 × width and ≈ 0.50 × height of the canvas |
|
||||
|
||||
Why icon-only + transparent: cards render on both light and dark backgrounds, so a baked-in
|
||||
(usually black) wordmark vanishes on dark. `deepseek.png` / `ernie.png` are 940×525 RGBA
|
||||
exemplars — eyeball your output against them.
|
||||
|
||||
## 1. Get the source
|
||||
|
||||
Ask the user for the brand logo (SVG preferred → crisp + already transparent; else a high-res
|
||||
transparent PNG, or a link to the official press/brand asset). Prefer an **icon-only** source;
|
||||
if they only have a full lockup, ask them to crop the glyph, or crop it yourself.
|
||||
|
||||
If the user **pasted** an image inline, it may not be on disk — recover the base64 `image`
|
||||
block from the session transcript (`~/.claude/projects/<slug>/*.jsonl`) and decode it to a file.
|
||||
|
||||
## 2. Generate (Pillow)
|
||||
|
||||
There's no system Pillow — use a venv:
|
||||
|
||||
```bash
|
||||
python3 -m venv /tmp/logo-venv && /tmp/logo-venv/bin/pip install -q Pillow
|
||||
```
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
src = Image.open("SOURCE").convert("RGBA") # icon-only, already transparent
|
||||
W, H = 940, 525
|
||||
target_h = round(H * 0.50) # glyph ≈ half the canvas height
|
||||
scale = target_h / src.height
|
||||
glyph = src.resize((round(src.width * scale), target_h), Image.LANCZOS)
|
||||
canvas = Image.new("RGBA", (W, H), (0, 0, 0, 0)) # transparent
|
||||
canvas.paste(glyph, ((W - glyph.width) // 2, (H - glyph.height) // 2), glyph)
|
||||
canvas.save("docs_new/cards/logos/<org-slug>.png")
|
||||
```
|
||||
|
||||
Notes:
|
||||
- **Wordmark present?** Crop to the glyph first (or ask the user for an icon-only asset). Don't
|
||||
ship text in the logo.
|
||||
- **Solid background?** Don't auto-strip it (risky) — ask the user for a transparent source.
|
||||
- **SVG source?** Rasterize at high res first (`cairosvg` / `rsvg-convert`), then run the above.
|
||||
- If the glyph is much wider than tall, cap by width instead (≈ 0.33 × W) so it doesn't overflow.
|
||||
|
||||
## 3. Verify
|
||||
|
||||
```bash
|
||||
sips -g pixelWidth -g pixelHeight -g hasAlpha docs_new/cards/logos/<org-slug>.png
|
||||
# → pixelWidth: 940 pixelHeight: 525 hasAlpha: yes
|
||||
```
|
||||
|
||||
## 4. Wire + track + validate
|
||||
|
||||
```bash
|
||||
# Card in the landing grid (keep card order aligned with the docs.json nav order):
|
||||
# <Card title="<NavGroup>" mode="card"
|
||||
# href="/cookbook/<category>/<Vendor>/<Model>"
|
||||
# img="/cards/logos/<org-slug>.png" />
|
||||
git add -f docs_new/cards/logos/<org-slug>.png # root .gitignore ignores *.png repo-wide
|
||||
cd docs_new && mint validate && mint broken-links # confirms the card href + img resolve
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
|
||||
// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>-benchmarks.jsx and
|
||||
// fill measured numbers — OR delete this file entirely if you have none yet (the
|
||||
// MDX simply omits the `benchmarks` import/prop).
|
||||
//
|
||||
// One entry per cell `match` tuple (same 5 keys as config cells). The card stays
|
||||
// "pending" until an entry has a non-null speed metric or accuracy. Speed shape:
|
||||
// speed: [{ workload: {dataset, isl, osl, max_concurrency}, ttft_ms, tpot_ms,
|
||||
// tokens_per_sec_per_gpu }, ...] // interactivity is derived (1000/TPOT)
|
||||
// Per-cell `accuracy: { <key>: <pct> }` overrides the config's defaultAccuracy.
|
||||
|
||||
export const benchmarks = [
|
||||
// EXAMPLE — one filled entry showing the shape; replace numbers, add one per cell.
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.0.0", // TODO: ASK the user for the sglang version these numbers were measured on — don't invent one
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
|
||||
],
|
||||
},
|
||||
// Bare-match stubs (no data yet) are fine — the card shows "pending" for these.
|
||||
{ match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "h100", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "single" } },
|
||||
{ match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" } },
|
||||
{ match: { hw: "b200", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" } },
|
||||
];
|
||||
@@ -0,0 +1,374 @@
|
||||
// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
|
||||
// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx, then:
|
||||
// 1. replace every __TOKEN__,
|
||||
// 2. fill cells[] with your verified recipes (the examples below show the shape),
|
||||
// 3. DELETE the hardware / playground axes / quantizations your model lacks.
|
||||
//
|
||||
// Instantiation tokens (skill fills these; distinct from the engine's runtime
|
||||
// {{PLACEHOLDER}} which MUST survive verbatim into the output):
|
||||
// __MODEL_DISPLAY__ display name, e.g. "DeepSeek-V4"
|
||||
// __MODEL_SLUG__ file slug, e.g. "deepseek-v4"
|
||||
// __HF_ORG__ HuggingFace org, e.g. "deepseek-ai"
|
||||
// __HF_REPO__ HuggingFace repo, e.g. "DeepSeek-V4-Flash"
|
||||
// __REASONING_PARSER__ e.g. "deepseek-v4" (delete the parsers axis if none)
|
||||
// __TOOLCALL_PARSER__ e.g. "deepseekv4" (delete the parsers axis if none)
|
||||
//
|
||||
// Mintlify: single `export const config = {...}` literal — no spreads/calls/IIFE,
|
||||
// no `!(x in y)`. Cells are denormalized: no --nnodes/--node-rank/--dist-init-addr/
|
||||
// --host/--port literals (the engine injects them).
|
||||
|
||||
export const config = {
|
||||
modelName: "__MODEL_DISPLAY__",
|
||||
|
||||
// List ONLY hardware you ship a cell for; unlisted ids auto-grey-out. The full
|
||||
// catalog is below — delete the families your model doesn't support (e.g. drop
|
||||
// every `mi*` if there's no AMD recipe).
|
||||
supportedHardware: [
|
||||
"h100", "h200", "b200", "b300", "gb200", "gb300",
|
||||
"mi300x", "mi325x", "mi350x", "mi355x",
|
||||
],
|
||||
|
||||
// OPTIONAL — declare GPUs the shared HARDWARE_CATALOG (in _deployment.jsx) doesn't
|
||||
// carry (workstation / desktop / future chips). The engine merges these in, so a
|
||||
// model-specific GPU is config data, never an engine-catalog edit. Add the id to
|
||||
// supportedHardware above too. Delete if you only use catalog GPUs.
|
||||
// hardware: [
|
||||
// { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" },
|
||||
// ],
|
||||
|
||||
// 2nd dim. Single-element `default` if the model has no variant axis; else list
|
||||
// real variants (e.g. {id:"flash",...},{id:"pro",...}) and key modelNames/
|
||||
// defaultAccuracy by them.
|
||||
variants: [
|
||||
{ id: "default", label: "Default" },
|
||||
],
|
||||
// 3rd dim. Keep only what your model ships (BF16 / FP8 / FP4 / …).
|
||||
quantizations: [
|
||||
{ id: "bf16", label: "BF16" },
|
||||
{ id: "fp8", label: "FP8" },
|
||||
{ id: "fp4", label: "FP4" },
|
||||
],
|
||||
strategies: [
|
||||
{ id: "low-latency", label: "Low-Latency" },
|
||||
{ id: "balanced", label: "Balanced" },
|
||||
{ id: "high-throughput", label: "High-Throughput" },
|
||||
],
|
||||
// `multi-N` id carries the node count for `--nnodes N`.
|
||||
nodesOptions: [
|
||||
{ id: "single", label: "Single Node" },
|
||||
{ id: "multi-2", label: "Multi-Nodes" },
|
||||
],
|
||||
|
||||
// HF slug lookup. Key by `variant|quant` (or `hw|variant|quant` for a per-hw
|
||||
// repackaging, e.g. an FP8 conversion only valid on one platform).
|
||||
modelNames: {
|
||||
"default|bf16": "__HF_ORG__/__HF_REPO__",
|
||||
"default|fp8": "__HF_ORG__/__HF_REPO__",
|
||||
"default|fp4": "__HF_ORG__/__HF_REPO__",
|
||||
},
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
NODE0_IP: { target: "command", label: "Head node IP", default: "<node0-ip>" },
|
||||
NODE_RANK: { target: "command", label: "This node rank", default: "<node-rank>" },
|
||||
HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
|
||||
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||
},
|
||||
|
||||
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||
-H 'Content-Type: application/json' \\
|
||||
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
|
||||
|
||||
// OPTIONAL — powers the benchmark card's "⚡ Reproduce" modal. Delete the whole
|
||||
// block (and the benchmarks file) if you have no measured numbers yet.
|
||||
benchmarkCommands: {
|
||||
speed:
|
||||
`python3 -m sglang.bench_serving \\
|
||||
--backend sglang \\
|
||||
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||
--model {{MODEL_NAME}} \\
|
||||
--dataset-name {{DATASET}} \\
|
||||
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
|
||||
// One entry per accuracy field. A value is a string, OR a {[variant]: string}
|
||||
// object when the command differs per variant. Keys must match ACCURACY_LABELS
|
||||
// in _deployment.jsx + the per-cell/defaultAccuracy keys.
|
||||
accuracy: {
|
||||
gsm8k_pct:
|
||||
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
|
||||
sgl-eval run gsm8k \\
|
||||
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
|
||||
--num-threads 32`,
|
||||
},
|
||||
// {{NUM_PROMPTS}} fallback per concurrency (else max(c*2, 200)).
|
||||
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 },
|
||||
},
|
||||
|
||||
// OPTIONAL — per-variant accuracy applied to EVERY cell of a variant (hardware-
|
||||
// independent, e.g. GPQA/AIME). Per-cell `accuracy` overrides. Keys must match
|
||||
// ACCURACY_LABELS + benchmarkCommands.accuracy. Delete if no numbers yet.
|
||||
defaultAccuracy: {
|
||||
default: { gsm8k_pct: null },
|
||||
},
|
||||
|
||||
// OPTIONAL — `# ...` hint lines prepended to multi-node commands, ONLY for hw
|
||||
// whose fabric needs manual NIC env (e.g. gb200 NVL72/MNNVL). NOT every multi-N
|
||||
// hw needs this — standard-IB DeepEP / Marlin multi-node don't. Delete if unused.
|
||||
multiNodeHints: {
|
||||
gb200: [
|
||||
"The following env vars may be needed depending on your cluster:",
|
||||
" GLOO_SOCKET_IFNAME=<your-nic>",
|
||||
" NVSHMEM_ENABLE_NIC_PE_MAPPING=1",
|
||||
" NVSHMEM_HCA_LIST=<your-hca-list>",
|
||||
],
|
||||
},
|
||||
|
||||
// Per-hw image for `docker run` framing. ASK the user which sglang build the recipes ran
|
||||
// on; don't guess a supporting release. Default below is :dev (nightly) — replace the tag
|
||||
// with the user's release if they give one. NVIDIA share one image; AMD uses ROCm tags.
|
||||
// GB200/GB300/B300 may need a `-cu130` (CUDA 13) tag — confirm per release.
|
||||
dockerImages: {
|
||||
h100: "lmsysorg/sglang:dev",
|
||||
h200: "lmsysorg/sglang:dev",
|
||||
b200: "lmsysorg/sglang:dev",
|
||||
b300: "lmsysorg/sglang:dev",
|
||||
gb200: "lmsysorg/sglang:dev",
|
||||
gb300: "lmsysorg/sglang:dev",
|
||||
mi300x: "lmsysorg/sglang:dev-rocm720-mi30x",
|
||||
mi325x: "lmsysorg/sglang:dev-rocm720-mi30x",
|
||||
mi350x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
||||
mi355x: "lmsysorg/sglang:dev-rocm720-mi35x",
|
||||
},
|
||||
|
||||
// Pre-selects the issue template's `model` dropdown on "Submit verified cell".
|
||||
// Must match that dropdown's value (usually `<hf-org>/<model-slug>`).
|
||||
github: {
|
||||
cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
|
||||
},
|
||||
|
||||
// Opt-in per axis. DELETE any axis your model doesn't expose (don't leave a stub).
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
|
||||
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
null, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
{ id: "cp", label: "CP", values: [null, 1, 2, 4] },
|
||||
{ id: "dpAttn", label: "DP-Attention",
|
||||
values: [
|
||||
null, false, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
],
|
||||
labels: { "auto": "Auto", "false": "Off" } },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "MoE Parallelism" ----- KEEP if MoE + multiple `--moe-*-backend`
|
||||
// choices. DELETE for dense models.
|
||||
moe: {
|
||||
backend: {
|
||||
options: [
|
||||
{ id: null, label: "Inherited" },
|
||||
{ id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] },
|
||||
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"] },
|
||||
{ id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] },
|
||||
],
|
||||
},
|
||||
ep: { label: "EP", values: [
|
||||
null, 1, 2, 4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
},
|
||||
|
||||
// ----- Card: "Parsers" ----- KEEP if the model has reasoning / tool-call
|
||||
// parsers (set the slugs below). DELETE the axis if neither applies.
|
||||
parsers: {
|
||||
items: [
|
||||
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser __REASONING_PARSER__" },
|
||||
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser __TOOLCALL_PARSER__" },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "Speculative Decoding" ----- KEEP if the model has spec-decoding
|
||||
// presets. Drop options the model doesn't support.
|
||||
speculative: {
|
||||
options: [
|
||||
{ id: "current", label: "Inherited from base" },
|
||||
{ id: "off", label: "Off (greedy)" },
|
||||
{ id: "mtp", label: "EAGLE / MTP",
|
||||
flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
|
||||
{ id: "ngram", label: "NGRAM",
|
||||
flags: ["--speculative-algorithm NGRAM",
|
||||
"--speculative-num-draft-tokens 16",
|
||||
"--speculative-ngram-max-bfs-breadth 10"],
|
||||
disable: { dpAttnOn: [true] },
|
||||
disableReason: "NGRAM is incompatible with DP-Attention. Turn DP-Attention off in the Attention card above to use NGRAM." },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "PD Disaggregation" ----- KEEP if the model supports prefill/
|
||||
// decode disaggregation. Delete `router` if you have no router topology.
|
||||
pdDisagg: {
|
||||
modes: [
|
||||
{ id: "off", label: "Off" },
|
||||
{ id: "prefill", label: "Prefill role" },
|
||||
{ id: "decode", label: "Decode role" },
|
||||
],
|
||||
transferBackends: [
|
||||
{ id: "mooncake", label: "Mooncake",
|
||||
env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"],
|
||||
envWhen: { hw: ["gb200", "gb300"] } },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
// `auto` is a sentinel (emits no --disaggregation-ib-device flag).
|
||||
ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
|
||||
// Router fronting prefill + decode; substitute <prefill-host>/<decode-host>.
|
||||
router: {
|
||||
port: 8000,
|
||||
command:
|
||||
`python3 -m sglang_router.launch_router \\
|
||||
--pd-disaggregation \\
|
||||
--prefill http://<prefill-host>:30000 \\
|
||||
--decode http://<decode-host>:30001 \\
|
||||
--host 0.0.0.0 --port 8000 \\
|
||||
--disable-circuit-breaker \\
|
||||
--health-check-interval-secs 999999`,
|
||||
},
|
||||
},
|
||||
|
||||
// ----- Card: "Hierarchical KV Cache" ----- KEEP if the model is large enough
|
||||
// that hierarchical KV caching matters.
|
||||
hicache: {
|
||||
backends: [
|
||||
{ id: null, label: "Auto" },
|
||||
{ id: "file", label: "File" },
|
||||
{ id: "mooncake", label: "Mooncake" },
|
||||
{ id: "hf3fs", label: "HF3FS" },
|
||||
{ id: "nixl", label: "NiXL" },
|
||||
],
|
||||
writePolicies: [
|
||||
{ id: "auto", label: "Auto" },
|
||||
{ id: "write_through", label: "Write-through" },
|
||||
{ id: "write_back", label: "Write-back" },
|
||||
{ id: "write_through_selective", label: "Write-through (selective)" },
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "HiSparse" ----- KEEP only for DSA-style sparse-attention models
|
||||
// (DeepSeek-V3.2/V4, GLM-5). Decode-only: shown when live PD-Disagg mode is `decode`.
|
||||
hisparse: {
|
||||
requiredFlags: ["--disable-radix-cache"],
|
||||
config: { top_k: 2048, device_buffer_size: 6144 },
|
||||
hostRatios: [
|
||||
{ id: 5, label: "5 (~1TB host)" },
|
||||
{ id: 10, label: "10 (~2TB host)" },
|
||||
],
|
||||
defaultHostRatio: 10,
|
||||
},
|
||||
|
||||
// ----- Card: "MegaMoE" ----- KEEP only for Blackwell MoE kernel-fusion models.
|
||||
megamoe: {
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"],
|
||||
excludesStrategy: ["low-latency", "balanced"],
|
||||
stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
|
||||
options: [
|
||||
{ id: "disabled", label: "Disabled" },
|
||||
{ id: "w4a8", label: "W4A8",
|
||||
flags: ["--moe-a2a-backend megamoe"],
|
||||
env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
// EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
|
||||
// your model's verified recipe, or DELETE families you don't support. `match`
|
||||
// MUST have exactly the 5 keys; env/flags are flat literals.
|
||||
cells: [
|
||||
// ==== NVIDIA Blackwell + FP4 (single node) ====
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
verified: true, // EXAMPLE — set false / replace with your verified recipe
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--moe-runner-backend flashinfer_mxfp4",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== NVIDIA Hopper + FP8 (single node, DP-attention + DeepEP) ====
|
||||
{
|
||||
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 4",
|
||||
"--dp 4",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== NVIDIA Hopper + FP4 (single node, Marlin W4A16 — Hopper has no FP4 runner) ====
|
||||
{
|
||||
match: { hw: "h100", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--moe-runner-backend marlin",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== AMD + BF16 (single node) — Triton attention + AITER; EP == TP for MoE ====
|
||||
{
|
||||
match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||
verified: true, // EXAMPLE
|
||||
env: ["SGLANG_USE_AITER=1", "SGLANG_ROCM_FUSED_DECODE_MLA=0"],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 8",
|
||||
"--ep 8",
|
||||
"--attention-backend triton",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ==== Multi-node example (2 nodes, TP=16) — engine injects --nnodes/--node-rank/
|
||||
// --dist-init-addr from match.nodes; do NOT add them here. ====
|
||||
{
|
||||
match: { hw: "b200", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" },
|
||||
verified: true, // EXAMPLE
|
||||
env: [],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 16",
|
||||
"--dp 16",
|
||||
"--enable-dp-attention",
|
||||
"--moe-a2a-backend deepep",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: __MODEL_DISPLAY__
|
||||
description: "__ONE_LINER__"
|
||||
tag: NEW
|
||||
mode: wide
|
||||
---
|
||||
|
||||
{/* TEMPLATE — instantiate via the cookbook-add-model skill, then DELETE this banner.
|
||||
(Frontmatter MUST stay the first thing in the file, so this note lives below it.)
|
||||
Replace every __TOKEN__, fill the TODO prose, delete the §3 subsections your model
|
||||
lacks. Tokens: __MODEL_DISPLAY__ __ONE_LINER__ __HF_ORG__ __MODEL_SLUG__ __HF_REPO__
|
||||
__REASONING_PARSER__ __TOOLCALL_PARSER__. MDX rules (JSX tables, labeled fences, no
|
||||
Docusaurus/@site/GitHub-alert/pipe-tables):
|
||||
.claude/skills/cookbook-add-model/references/mintlify-authoring.md */}
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
{/* TODO: 1-2 paragraph intro from the HF card — what the model is, release date,
|
||||
license, architecture highlights, context length. Keep it lean. */}
|
||||
**__MODEL_DISPLAY__** is __ONE_LINER__.
|
||||
|
||||
{/* TODO: variants table (JSX, NOT a markdown pipe table). Drop the table if there's
|
||||
a single variant and inline the HF link in the intro paragraph above instead. */}
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/__HF_ORG__/__HF_REPO__">__MODEL_DISPLAY__</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>TODO</td>
|
||||
<td style={{padding: "9px 12px"}}>TODO</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** {/* TODO e.g. `temperature=1.0`, `top_p=1.0` (informational; do NOT hardcode in sample code) */}
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/__HF_ORG__/__HF_REPO__).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
{/* TODO: model/hardware-specific tuning notes, caveats, known issues. Delete if none. */}
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
{/* Keep only the subsections that apply. Each runnable block is followed by an
|
||||
**Output Example:** + a ```text Output block with REAL server output. */}
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer.
|
||||
|
||||
{/* This example assumes a SEPARATE-FIELD parser (thinking → `reasoning_content`,
|
||||
answer → `content`). If your parser emits inline `<think>...</think>` tags inside
|
||||
`content`, parse the tags from `content` instead. */}
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="__HF_ORG__/__HF_REPO__",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
TODO: paste real server output here.
|
||||
```
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
{/* TODO: tool-calling example + **Output Example:**. On thinking-mode models the
|
||||
follow-up may put text in `reasoning_content`; print both that and `content`. */}
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
{/* TODO: keep only if the model is large enough for hierarchical KV caching; link
|
||||
the HiCache card in the Playground. Otherwise delete this subsection. */}
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
name: cookbook-review-pr
|
||||
description: Review a pull request against the SGLang Cookbook (docs_new/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
|
||||
---
|
||||
|
||||
# Cookbook Review PR
|
||||
|
||||
Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
|
||||
shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
|
||||
model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
|
||||
plus an MDX page. This checklist targets that layout. Field-schema detail lives in
|
||||
`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
|
||||
than restating.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/cookbook-review-pr <PR number>
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
|
||||
2. `gh pr diff <N> --repo sgl-project/sglang`
|
||||
3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
|
||||
4. Run every checklist item against the diff.
|
||||
5. Output per-file verdicts + overall recommendation.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. File hygiene
|
||||
- A cookbook PR should only touch: `docs_new/src/snippets/configs/<vendor>/*.jsx`
|
||||
(config + benchmarks), `docs_new/cookbook/**/*.mdx`, `docs_new/docs.json`,
|
||||
`docs_new/cookbook/<category>/intro.mdx` (vendor card), `docs_new/cards/logos/<vendor>.png`
|
||||
(new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
|
||||
- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
|
||||
for unrelated commits accidentally included.
|
||||
- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
|
||||
model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
|
||||
(see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
|
||||
|
||||
### 2. Config quality (the per-model `config`)
|
||||
- Single `export const config = { ... }` literal — **no** function calls, spreads,
|
||||
fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
|
||||
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback).
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
- `github.cookbookModel` matches the issue-template `model` dropdown value.
|
||||
- `playgroundFeatures` axes are pruned to what the model supports — no empty/stub axes
|
||||
(`megamoe` only on Blackwell MoE; `hisparse` only DSA-style; `pdDisagg.router` only with
|
||||
a PD topology).
|
||||
- **No leftover `__TOKEN__`** — the config was stamped from the template and every
|
||||
placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
|
||||
nothing).
|
||||
- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
|
||||
greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
|
||||
|
||||
### 3. Cells / 5-dim matrix
|
||||
- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
|
||||
- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
|
||||
`commonFlags` reference (Mintlify won't inline it).
|
||||
- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
|
||||
(the renderer injects them from `match.nodes`).
|
||||
- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
|
||||
- Flag order: `--model-path` first, then parallelism, then MoE, then tuning, `--host`/`--port`
|
||||
last (the playground's insert anchors assume this).
|
||||
- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
|
||||
(BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
|
||||
|
||||
### 4. Benchmarks
|
||||
- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
|
||||
- `defaultAccuracy` keys ∈ `ACCURACY_LABELS` (and `benchmarkCommands.accuracy`).
|
||||
- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
|
||||
that only released FP8/FP4 is a factual bug.
|
||||
- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
|
||||
from the `sglang serve` deploy command.
|
||||
- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
|
||||
guessed/placeholder value (no leftover `0.0.0`).
|
||||
|
||||
### 5. Doc ↔ config parity (the #1 finding)
|
||||
- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
|
||||
equal what the engine emits from the corresponding cell — same flags, same order. Drift
|
||||
here is the most common review miss.
|
||||
|
||||
### 6. Commands / port
|
||||
- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
|
||||
`python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
|
||||
guard against prose/cells reintroducing the old launcher.
|
||||
- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
|
||||
Launch port must match client/curl port on the same page.
|
||||
|
||||
### 7. Frontmatter
|
||||
- Every new MDX page has `title:` and `metatags.description:` (a real one-line value prop,
|
||||
not copied from another vendor).
|
||||
- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
|
||||
pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs_new/cookbook/`).
|
||||
- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
|
||||
- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
|
||||
"↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
|
||||
|
||||
### 8. Navigation & homepage
|
||||
- New page → `docs_new/docs.json` updated: under the right vendor group inside
|
||||
`navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
|
||||
`cookbook/<category>/<Vendor>/<Model>`.
|
||||
- Homepage `<Card href>` in `docs_new/cookbook/<category>/intro.mdx` points to the vendor's
|
||||
flagship; new vendors get a new `<Card>` + a logo at `docs_new/cards/logos/<vendor>.png` —
|
||||
**940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
|
||||
`git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
|
||||
- Don't change `docs_new/cookbook/intro.mdx` for individual model adds (top-level only).
|
||||
|
||||
### 9. Links & factual
|
||||
- HuggingFace URLs resolve to a real model. License section matches the actual HF license
|
||||
(don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
|
||||
The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
|
||||
guessed version.
|
||||
- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
|
||||
or `../`-relative links. `docs.sglang.io` is canonical.
|
||||
- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
|
||||
not `${VAR}` (a bash no-op).
|
||||
|
||||
### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
|
||||
- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
|
||||
blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
|
||||
or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
|
||||
- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
|
||||
the opening fence); a fenced block nested inside another uses four backticks outside.
|
||||
- Every runnable invocation block is followed by `**Output Example:**` + a `text Output`
|
||||
fenced block (real output, or `Pending update...` only with user acknowledgement).
|
||||
- Reasoning-parser example matches the parser's **output shape**: separate-field
|
||||
(`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
|
||||
- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
|
||||
`generation_config.json` defaults); listing them in §1 informationally is fine.
|
||||
|
||||
### 10. Quantization rules
|
||||
- FP4 is Blackwell-only (B200/B300/GB300) — never AMD; AMD FP4 chips must be `disabled`.
|
||||
- BF16 / FP8 work on NVIDIA and AMD. FP8 configs adding `--kv-cache-dtype fp8_e4m3` should
|
||||
note the accuracy trade-off.
|
||||
|
||||
### 11. Scope
|
||||
- Changes match the PR title. Flag global changes hiding behind a platform-specific title
|
||||
(e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
|
||||
the PR body.
|
||||
|
||||
### 12. Duplicate PRs
|
||||
- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
|
||||
risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
|
||||
|
||||
### 13. Build / validate
|
||||
```bash
|
||||
cd docs_new
|
||||
mint validate
|
||||
mint broken-links
|
||||
```
|
||||
Optional: `mint dev` for a visual smoke test.
|
||||
|
||||
### 14. Reviewer feedback
|
||||
- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
|
||||
addressed? Unresolved requested-changes should be flagged.
|
||||
|
||||
### 15. Grammar & spelling
|
||||
- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
|
||||
Flag each with the exact wrong text + correction.
|
||||
|
||||
## Output
|
||||
|
||||
Per file:
|
||||
- ✅ PASS
|
||||
- ⚠️ ISSUE: \<what\>
|
||||
- 🔴 BLOCK: \<what\>
|
||||
|
||||
Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**
|
||||
@@ -0,0 +1,109 @@
|
||||
name: 🧪 Playground - Verified Cell Submission
|
||||
description: Submit a deployment recipe you verified in the cookbook Playground. A maintainer will review and promote it into the cell catalog via PR.
|
||||
title: "[Playground] Verified cell: <fill from playground>"
|
||||
labels: ["cookbook", "playground-submission"]
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for verifying a deployment recipe!
|
||||
|
||||
The cookbook **Playground** section of each model page pre-fills this
|
||||
form. When your overrides diverge from the verified cell, click
|
||||
**Submit ↗**, confirm the SGLang version / benchmark / notes in the
|
||||
**Submit verified cell** dialog and tick the attestations, then
|
||||
**Open submission on GitHub →**. The model, combination, proposed cell
|
||||
snippet, and existing cell are filled automatically — please double-check
|
||||
the snippet matches what you ran.
|
||||
|
||||
Once a maintainer with access to the listed hardware reproduces the
|
||||
recipe, they'll convert this issue into a PR against the cookbook cell
|
||||
catalog and close this issue.
|
||||
|
||||
- type: dropdown
|
||||
id: model
|
||||
attributes:
|
||||
label: Cookbook model
|
||||
description: Which model's cookbook does this cell belong to?
|
||||
options:
|
||||
- deepseek-ai/deepseek-v4
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: combination
|
||||
attributes:
|
||||
label: Combination
|
||||
description: "Format: hw / variant / quant / strategy / nodes (auto-filled by the playground)."
|
||||
placeholder: "b200 / flash / fp4 / low-latency / single"
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: cell-snippet
|
||||
attributes:
|
||||
label: Proposed cell snippet
|
||||
description: |
|
||||
The cell object exactly as it should land in the `cells: [...]` array
|
||||
of the cookbook config. Auto-generated by the playground — please do not
|
||||
hand-edit unless you're sure.
|
||||
render: javascript
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: existing-cell
|
||||
attributes:
|
||||
label: Existing cell at this match (for diff)
|
||||
description: |
|
||||
The current verified cell at the same `match` tuple, if any. The
|
||||
playground fills this so the maintainer can see exactly what changed.
|
||||
Leave empty if no cell yet exists at this match.
|
||||
render: javascript
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: input
|
||||
id: sglang-version
|
||||
attributes:
|
||||
label: SGLang version
|
||||
description: Version, tag, or git SHA you tested against.
|
||||
placeholder: "sglang==0.5.4 (or git SHA abc1234)"
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: bench-result
|
||||
attributes:
|
||||
label: Benchmark result (optional but encouraged)
|
||||
description: |
|
||||
One-line perf numbers (TTFT, TPOT, tokens/sec, accepted-rate, etc.).
|
||||
Helps maintainers decide whether this recipe should replace the
|
||||
existing one or be added as an alternative.
|
||||
placeholder: "TTFT 95 ms / TPOT 18 ms / 1820 tok/s @ bs=64"
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: notes
|
||||
attributes:
|
||||
label: Notes / caveats
|
||||
description: |
|
||||
Anything unusual: cluster config, env-var quirks, NIC mappings,
|
||||
multi-node bootstrap details, etc.
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: checkboxes
|
||||
id: attestation
|
||||
attributes:
|
||||
label: Attestation
|
||||
description: All three must be ticked. We trust contributors but maintainers will still re-verify before merging.
|
||||
options:
|
||||
- label: I ran this exact command on the listed hardware.
|
||||
required: true
|
||||
- label: The server reached READY and answered a cURL request successfully.
|
||||
required: true
|
||||
- label: Output looked correct on at least one prompt.
|
||||
required: true
|
||||
@@ -1,10 +1,104 @@
|
||||
---
|
||||
title: DeepSeek-V4
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V4 with SGLang — a next-generation MoE model from DeepSeek."
|
||||
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for the Flash (284B) and Pro (1.6T) Mixture-of-Experts models."
|
||||
tag: NEW
|
||||
mode: wide
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200). For **RTX PRO 6000 (SM120)**, use the nightly `lmsysorg/sglang:dev` instead — SM120 support isn't in `:latest` yet (see the RTX PRO 6000 note below).
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
The knobs come in two flavors:
|
||||
|
||||
- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
|
||||
- **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only).
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. It ships as two Instruct repos (one per variant) plus matching Base repos:
|
||||
@@ -29,76 +123,26 @@ tag: NEW
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving: B200 / GB200 / GB300 / H200 on 4 GPUs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); H100 (TP=8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>49B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 8 GPU / GB200 8 GPU (2 nodes) / GB300 4 GPU / H200 8 GPU (FP4) or 16 GPU (SGLang FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Instruct repos ship **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers all GPUs that support FP4). The Base (pre-trained only) variants — `DeepSeek-V4-Flash-Base`, `DeepSeek-V4-Pro-Base` — ship pure FP8 mixed and are **not** for chat / tool calling.
|
||||
Both Instruct repos ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.
|
||||
|
||||
**Key Features** (per the official model card):
|
||||
**Highlights:** hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar.
|
||||
|
||||
- **Hybrid Attention Architecture** — combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) for long-context efficiency. At 1M-token context, DeepSeek-V4-Pro uses only ~27% of per-token inference FLOPs and ~10% of KV cache compared with DeepSeek-V3.2.
|
||||
- **Manifold-Constrained Hyper-Connections (mHC)** — strengthens residual connections, improving signal-propagation stability across layers while preserving expressivity.
|
||||
- **Muon optimizer** — faster convergence and greater training stability.
|
||||
- **Context length: 1M tokens**; pre-trained on 32T+ diverse, high-quality tokens.
|
||||
- **Three reasoning modes**: *Non-think* (fast, intuitive responses), *Think High* (conscious logical analysis, slower but more accurate), *Think Max* (push reasoning to its fullest extent). Recommend a ≥ 384K context window when running Think Max.
|
||||
- Ships with a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar (`<｜DSML｜tool_calls>` / `<｜DSML｜invoke>` / `<｜DSML｜parameter>`).
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=1.0`.
|
||||
|
||||
**Recommended Generation Parameters:** `temperature=1.0`, `top_p=1.0` (per the official model card).
|
||||
**Resources:** HuggingFace · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
|
||||
|
||||
**License:** MIT.
|
||||
|
||||
**Resources:**
|
||||
|
||||
- HuggingFace: [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)
|
||||
- ModelScope: [DeepSeek-V4-Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash), [DeepSeek-V4-Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. Choose based on your hardware platform.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Image:** Use `lmsysorg/sglang:latest` for all supported hardware platforms (B300 / B200 / GB200 / GB300 / H200 / H100).
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to actually launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
SGLang supports three main serving recipes for DeepSeek-V4 with different latency/throughput trade-offs (`low-latency`, `balanced`, `max-throughput`), plus specialized recipes for long-context (`cp`, prefill context-parallel) and prefill/decode disaggregation (`pd-disagg`). The interactive generator below emits the exact launch command for any `(hardware, variant, recipe)` combination.
|
||||
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware + recipe combination.
|
||||
|
||||
import { DeepSeekV4Deployment } from "/src/snippets/autoregressive/deepseek-v4-deployment.jsx";
|
||||
|
||||
<DeepSeekV4Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
## 2. Configuration Tips
|
||||
|
||||
{/* TODO: expand this section as more recipes are validated end-to-end. */}
|
||||
|
||||
@@ -112,8 +156,8 @@ The generator currently picks values on the **conservative** side (mirroring an
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
|
||||
- `max-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP currently requires `SGLANG_ENABLE_SPEC_V2=1`.
|
||||
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP runs on the v2 speculative path (`SGLANG_ENABLE_SPEC_V2`, enabled by default).
|
||||
|
||||
**EPLB + DeepEP Waterfill (Experimental)**
|
||||
|
||||
@@ -161,8 +205,9 @@ requires `--moe-a2a-backend deepep`.
|
||||
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
|
||||
|
||||
```bash Command
|
||||
# Please use latest main branch for this feature
|
||||
sglang serve deepseek-ai/DeepSeek-V4-Flash \
|
||||
# Please use the latest main branch for this feature.
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash \
|
||||
--tp 4 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--enable-deepseek-v4-fp4-indexer
|
||||
@@ -170,25 +215,31 @@ sglang serve deepseek-ai/DeepSeek-V4-Flash \
|
||||
|
||||
<a id="hopper-note" />
|
||||
|
||||
**Hopper (H200) note**
|
||||
**Hopper (H100 / H200) note**
|
||||
|
||||
We provide two different options for running DeepSeek-V4 models on Hopper devices (H200)
|
||||
- Original FP4 checkpoints: To run original FP4 checkpoints, we provide two different options for w4a16 MoE kernels: Marlin (`--moe-runner-backend marlin`) and Flashinfer (`--moe-runner-backend flashinfer_mxfp4`). For this variant we only support Tensor Parallelism. Complete Pro model can be run on a single H200 node with this option.
|
||||
- Converted FP8 checkpoints: We also provide pre-converted FP8 checkpoints (`sgl-project/DeepSeek-V4-Flash-FP8`, `sgl-project/DeepSeek-V4-Pro-FP8`), which support more parallelism and features.
|
||||
Two options are available for running DeepSeek-V4 on Hopper:
|
||||
|
||||
- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
|
||||
- **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).
|
||||
|
||||
PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1`
|
||||
(or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake
|
||||
can discover the IB HCAs; without IB exposure mooncake silently falls back to
|
||||
TCP, which can lead to garbled KV transfer on large checkpoints.
|
||||
|
||||
**RTX PRO 6000 (SM120 / Blackwell Desktop) note**
|
||||
|
||||
RTX PRO 6000 (96 GB) runs **Flash only** — V4-Pro doesn't fit on 8× 96 GB. It uses the
|
||||
**low-latency / TP-only** recipe (TP=4, single node) with the **Marlin** W4A16 MoE runner and
|
||||
`--mem-fraction-static 0.70`; the Deploy panel greys out the other recipes for this card.
|
||||
HiCache and MegaMoE are **not** supported on RTX PRO 6000. For Docker, use the nightly `lmsysorg/sglang:dev` image — SM120 support isn't in `lmsysorg/sglang:latest` yet (the Deploy panel's Docker mode already points this card at `:dev`).
|
||||
|
||||
**MegaMoE**
|
||||
|
||||
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
|
||||
on MoE layers. To enable it, use the **MegaMoE** toggle in the
|
||||
[command generator above](#3-model-deployment) — the generator will swap
|
||||
`--moe-a2a-backend deepep` for `--moe-a2a-backend megamoe` and add the
|
||||
relevant env vars automatically.
|
||||
|
||||
on MoE layers. To enable it, use the **MegaMoE** chip in the Playground
|
||||
below — the playground will swap `--moe-a2a-backend deepep` for
|
||||
`--moe-a2a-backend megamoe` and add the relevant env vars automatically.
|
||||
|
||||
Two variants are exposed:
|
||||
- **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations).
|
||||
@@ -198,8 +249,10 @@ Two variants are exposed:
|
||||
(~89.5 GPQA on Pro).
|
||||
|
||||
Notes:
|
||||
- MegaMoE is **not** supported on Hopper (H100 / H200) nor on the `low-latency` / `balanced` / `cp` settings — it is only wired into the `max-throughput` recipe on Blackwell. When running MegaMoE, don't set `--moe-runner-backend` manually.
|
||||
- Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space. (recommended: 8320 for max-throughput).
|
||||
- MegaMoE is **only supported on Blackwell GPUs** (B200 / B300 / GB200 / GB300). The chip is hidden when the Deploy panel's base cell sits on Hopper (H100 / H200).
|
||||
- MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it.
|
||||
- When running MegaMoE, don't set `--moe-runner-backend` manually.
|
||||
- Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).
|
||||
|
||||
**GB300 PD-Disagg cross-pod MNNVL**
|
||||
|
||||
@@ -208,32 +261,11 @@ fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If
|
||||
this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`
|
||||
to both prefill and decode `sglang serve` commands.
|
||||
|
||||
## 4. Model Invocation
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 4.1 Basic Usage
|
||||
### 3.1 Reasoning
|
||||
|
||||
For basic API usage and request examples, see:
|
||||
|
||||
- [Basic API Usage](../../../docs/basic_usage/send_request)
|
||||
|
||||
Once the server is running (for example via the command generator above), send a request:
|
||||
|
||||
```shell Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"messages": [{"role": "user", "content": "What is 15% of 240?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
> **PD-Disagg note**: if you deployed with the `pd-disagg` recipe from the generator above, the prefill server is on port `30000`, the decode server on `30001`, and the **router** on port `8000` — client traffic should target `http://localhost:8000`, not `:30000`.
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
Enable the `deepseek-v4` reasoning parser (check the box in the [command panel above](#3-model-deployment)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
<Accordion title="Streaming with Thinking Process (Python)">
|
||||
|
||||
@@ -305,9 +337,9 @@ Multiply the decimal form by 240:
|
||||
|
||||
</Accordion>
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `deepseekv4` tool-call parser (check the box in the [command panel above](#3-model-deployment)) to surface structured tool calls via `message.tool_calls`.
|
||||
Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
<Accordion title="Python Example with Thinking Process">
|
||||
|
||||
@@ -401,339 +433,15 @@ The user wants to know the weather in Beijing. I'll use the get_weather function
|
||||
|
||||
</Accordion>
|
||||
|
||||
#### 4.2.3 HiCache (Hierarchical KV Caching)
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.
|
||||
|
||||
To enable HiCache, use the **HiCache** toggle in the [command generator above](#3-model-deployment):
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**:
|
||||
|
||||
- **L2 (GPU + CPU):** Offloads cold KV pages to CPU memory. Enables `SGLANG_ENABLE_UNIFIED_RADIX_TREE=1` for intelligent hierarchical prefix caching.
|
||||
- **L3 (GPU + CPU + Storage):** Coming soon.
|
||||
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only.
|
||||
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
|
||||
|
||||
The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.
|
||||
|
||||
For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache).
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
For accuracy benchmarking on DeepSeek-V4 models, please make sure that:
|
||||
- `SGLANG_DEFAULT_THINKING=1 SGLANG_REASONING_EFFORT=max` are set when launching model.
|
||||
- For GPQA and AIME25 benchmarks, run at least 16 turns to reduce randomness.
|
||||
|
||||
#### 5.1.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
- DeepSeek-V4-Pro (FP4, B300, low-latency)
|
||||
```
|
||||
Accuracy: 0.965
|
||||
Invalid: 0.000
|
||||
```
|
||||
- DeepSeek-V4-Pro (FP4, H200, low-latency)
|
||||
```
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
#### 5.1.2 GPQA Diamond Benchmark
|
||||
|
||||
For GPQA Diamond benchmark, we recommend applying [sgl-eval](https://github.com/sgl-project/sgl-eval) as the benchmark tool.
|
||||
|
||||
```shell Command
|
||||
# Install
|
||||
pip install git+https://github.com/sgl-project/sgl-eval
|
||||
|
||||
# For Flash model, reference accuracy: 88.1%
|
||||
sgl-eval run gpqa --model deepseek-ai/DeepSeek-V4-Flash --api-key <api-key> --n-repeats 16 --max-tokens 200000 --temperature 1.0 --top-p 1.0 --thinking --out-dir /sgl-workspace/logs --base-url http://localhost:30000/v1
|
||||
|
||||
# For Pro model, reference accuracy: 90.1%
|
||||
sgl-eval run gpqa --model deepseek-ai/DeepSeek-V4-Pro --api-key <api-key> --n-repeats 16 --max-tokens 400000 --temperature 1.0 --top-p 1.0 --thinking --out-dir /sgl-workspace/logs --base-url http://localhost:30000/v1
|
||||
```
|
||||
|
||||
#### 5.1.3 AIME25 Benchmark
|
||||
|
||||
For AIME25 benchmark, we recommend applying [sgl-eval](https://github.com/sgl-project/sgl-eval) as the benchmark tool.
|
||||
|
||||
```shell Command
|
||||
# Install
|
||||
pip install git+https://github.com/sgl-project/sgl-eval
|
||||
|
||||
# For Flash model, reference accuracy: ~95%
|
||||
sgl-eval run aime25 --model deepseek-ai/DeepSeek-V4-Flash --api-key <api-key> --n-repeats 16 --max-tokens 200000 --temperature 1.0 --top-p 1.0 --thinking --out-dir /sgl-workspace/logs --base-url http://localhost:30000/v1
|
||||
|
||||
# For Pro model, reference accuracy: ~97.5%
|
||||
sgl-eval run aime25 --model deepseek-ai/DeepSeek-V4-Pro --api-key <api-key> --n-repeats 16 --max-tokens 400000 --temperature 1.0 --top-p 1.0 --thinking --out-dir /sgl-workspace/logs --base-url http://localhost:30000/v1
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
We use SGLang's built-in benchmarking tool with its `random` dataset — real prompts sampled from [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) and then truncated/padded to a controlled length. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.2.1 Hopper
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (4x)
|
||||
- Model: DeepSeek-V4-Flash (FP4)
|
||||
- Tensor Parallelism: 4
|
||||
- sglang version: 0.5.12
|
||||
|
||||
##### Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** H200 · DeepSeek-V4-Flash · FP4 · Low-Latency. See the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 15.98
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.63
|
||||
Input token throughput (tok/s): 381.86
|
||||
Output token throughput (tok/s): 264.13
|
||||
Peak output token throughput (tok/s): 324.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 645.98
|
||||
Concurrency: 1.00
|
||||
Accept length: 2.96
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1596.65
|
||||
Median E2E Latency (ms): 1274.48
|
||||
P90 E2E Latency (ms): 2950.70
|
||||
P99 E2E Latency (ms): 3333.18
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 147.26
|
||||
Median TTFT (ms): 132.22
|
||||
P99 TTFT (ms): 181.37
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.50
|
||||
Median TPOT (ms): 3.48
|
||||
P99 TPOT (ms): 4.18
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.44
|
||||
Median ITL (ms): 3.36
|
||||
P95 ITL (ms): 5.06
|
||||
P99 ITL (ms): 5.15
|
||||
Max ITL (ms): 35.31
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** H200 · DeepSeek-V4-Flash · FP4 · Max-Throughput. See the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 198.42
|
||||
Total input tokens: 512842
|
||||
Total input text tokens: 512842
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 510765
|
||||
Request throughput (req/s): 5.04
|
||||
Input token throughput (tok/s): 2584.65
|
||||
Output token throughput (tok/s): 2574.64
|
||||
Peak output token throughput (tok/s): 4400.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5159.28
|
||||
Concurrency: 96.21
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 19090.29
|
||||
Median E2E Latency (ms): 18328.71
|
||||
P90 E2E Latency (ms): 35698.68
|
||||
P99 E2E Latency (ms): 39161.43
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 302.41
|
||||
Median TTFT (ms): 131.35
|
||||
P99 TTFT (ms): 2172.03
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 37.46
|
||||
Median TPOT (ms): 37.72
|
||||
P99 TPOT (ms): 55.72
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 36.85
|
||||
Median ITL (ms): 21.75
|
||||
P95 ITL (ms): 107.64
|
||||
P99 ITL (ms): 134.58
|
||||
Max ITL (ms): 1930.74
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 Blackwell
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (4x)
|
||||
- Model: DeepSeek-V4-Flash (FP4)
|
||||
- Tensor Parallelism: 4
|
||||
- sglang version: 0.5.12
|
||||
|
||||
##### Latency-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** B200 · DeepSeek-V4-Flash · FP4 · Low-Latency. See the [command panel above](#3-model-deployment).
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 15.25
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.66
|
||||
Input token throughput (tok/s): 400.06
|
||||
Output token throughput (tok/s): 276.72
|
||||
Peak output token throughput (tok/s): 308.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 676.78
|
||||
Concurrency: 1.00
|
||||
Accept length: 2.73
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1523.83
|
||||
Median E2E Latency (ms): 1173.50
|
||||
P90 E2E Latency (ms): 2770.33
|
||||
P99 E2E Latency (ms): 3233.82
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 102.72
|
||||
Median TTFT (ms): 85.94
|
||||
P99 TTFT (ms): 134.79
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.40
|
||||
Median TPOT (ms): 3.42
|
||||
P99 TPOT (ms): 4.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.38
|
||||
Median ITL (ms): 3.06
|
||||
P95 ITL (ms): 4.60
|
||||
P99 ITL (ms): 4.95
|
||||
Max ITL (ms): 34.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### Throughput-Sensitive Benchmark
|
||||
|
||||
- **Model Deployment Command:** B200 · DeepSeek-V4-Flash · FP4 · Max-Throughput (MegaMoE W4A4). See the [command panel above](#3-model-deployment) — flip the **MegaMoE** toggle to **W4A4** to reproduce these numbers; the default Max-Throughput recipe uses `--moe-a2a-backend deepep` and runs slower.
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 105.10
|
||||
Total input tokens: 512842
|
||||
Total input text tokens: 512842
|
||||
Total generated tokens: 510855
|
||||
Total generated tokens (retokenized): 510682
|
||||
Request throughput (req/s): 9.51
|
||||
Input token throughput (tok/s): 4879.44
|
||||
Output token throughput (tok/s): 4860.54
|
||||
Peak output token throughput (tok/s): 6600.00
|
||||
Peak concurrent requests: 117
|
||||
Total token throughput (tok/s): 9739.98
|
||||
Concurrency: 94.34
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9915.50
|
||||
Median E2E Latency (ms): 9521.19
|
||||
P90 E2E Latency (ms): 17726.66
|
||||
P99 E2E Latency (ms): 24910.72
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 349.95
|
||||
Median TTFT (ms): 68.23
|
||||
P99 TTFT (ms): 4581.26
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 19.86
|
||||
Median TPOT (ms): 17.96
|
||||
P99 TPOT (ms): 61.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.76
|
||||
Median ITL (ms): 13.23
|
||||
P95 ITL (ms): 44.79
|
||||
P99 ITL (ms): 88.25
|
||||
Max ITL (ms): 2499.49
|
||||
==================================================
|
||||
```
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,306 @@
|
||||
// DeepSeek-V4 per-cell benchmark numbers, keyed by the same `match` tuple as
|
||||
// deepseek-v4.jsx cells. See _deployment.jsx for the speed/accuracy schema.
|
||||
// Measured on sglang v0.5.12.post1.
|
||||
export const benchmarks = [
|
||||
// ====================================================================
|
||||
// B200 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 428, tpot_ms: 3.53, tokens_per_sec_per_gpu: 44 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 3111, tpot_ms: 23.82, tokens_per_sec_per_gpu: 121 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 4228, tpot_ms: 60.98, tokens_per_sec_per_gpu: 225 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 4628, tpot_ms: 88.25, tokens_per_sec_per_gpu: 643 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 105918, tpot_ms: 70.73, tokens_per_sec_per_gpu: 881 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 273356, tpot_ms: 71.61, tokens_per_sec_per_gpu: 889 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2326, tpot_ms: 69.9, tokens_per_sec_per_gpu: 99 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 7242, tpot_ms: 152.09, tokens_per_sec_per_gpu: 192 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b200", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
},
|
||||
// ====================================================================
|
||||
// B300 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 205, tpot_ms: 3.43, tokens_per_sec_per_gpu: 54 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 1856, tpot_ms: 14.82, tokens_per_sec_per_gpu: 205 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2363, tpot_ms: 34.4, tokens_per_sec_per_gpu: 402 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 2812, tpot_ms: 51.65, tokens_per_sec_per_gpu: 1092 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 82556, tpot_ms: 55.37, tokens_per_sec_per_gpu: 1130 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 207987, tpot_ms: 54.05, tokens_per_sec_per_gpu: 1171 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 239, tpot_ms: 5.04, tokens_per_sec_per_gpu: 24 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 830, tpot_ms: 15.55, tokens_per_sec_per_gpu: 101 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 1866, tpot_ms: 54.48, tokens_per_sec_per_gpu: 139 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 6325, tpot_ms: 123.95, tokens_per_sec_per_gpu: 237 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "b300", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 99139, tpot_ms: 44.37, tokens_per_sec_per_gpu: 476 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 241544, tpot_ms: 43.51, tokens_per_sec_per_gpu: 492 },
|
||||
],
|
||||
},
|
||||
// ====================================================================
|
||||
// GB200 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 335, tpot_ms: 3.67, tokens_per_sec_per_gpu: 47 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 2440, tpot_ms: 15.95, tokens_per_sec_per_gpu: 163 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2560, tpot_ms: 39.71, tokens_per_sec_per_gpu: 342 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 3995, tpot_ms: 82.56, tokens_per_sec_per_gpu: 718 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 128397, tpot_ms: 84.95, tokens_per_sec_per_gpu: 757 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 330479, tpot_ms: 86.7, tokens_per_sec_per_gpu: 741 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "multi-2" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 343, tpot_ms: 6.47, tokens_per_sec_per_gpu: 18 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 1345, tpot_ms: 23.85, tokens_per_sec_per_gpu: 65 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "multi-2" },
|
||||
},
|
||||
{
|
||||
match: { hw: "gb200", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" },
|
||||
},
|
||||
// ====================================================================
|
||||
// GB300 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 380, tpot_ms: 4.4, tokens_per_sec_per_gpu: 38 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 2960, tpot_ms: 21.26, tokens_per_sec_per_gpu: 125 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 2671, tpot_ms: 45.88, tokens_per_sec_per_gpu: 299 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 4823, tpot_ms: 94.04, tokens_per_sec_per_gpu: 637 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 146954, tpot_ms: 97.24, tokens_per_sec_per_gpu: 662 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 368557, tpot_ms: 99.33, tokens_per_sec_per_gpu: 651 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 363, tpot_ms: 6.53, tokens_per_sec_per_gpu: 36 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 1275, tpot_ms: 20.75, tokens_per_sec_per_gpu: 152 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
},
|
||||
// ====================================================================
|
||||
// H200 + FP8
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp8", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 204, tpot_ms: 3.38, tokens_per_sec_per_gpu: 68 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 538, tpot_ms: 11.42, tokens_per_sec_per_gpu: 264 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
|
||||
ttft_ms: 738, tpot_ms: 36.27, tokens_per_sec_per_gpu: 385 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
|
||||
ttft_ms: 39806, tpot_ms: 80.13, tokens_per_sec_per_gpu: 393 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp8", strategy: "high-throughput", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
|
||||
ttft_ms: 195293, tpot_ms: 130.35, tokens_per_sec_per_gpu: 493 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 4096 },
|
||||
ttft_ms: 502615, tpot_ms: 130.31, tokens_per_sec_per_gpu: 490 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp8", strategy: "low-latency", nodes: "multi-2" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp8", strategy: "balanced", nodes: "multi-2" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp8", strategy: "high-throughput", nodes: "multi-2" },
|
||||
},
|
||||
// ====================================================================
|
||||
// H200 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
sglang_version: "0.5.12.post1",
|
||||
speed: [
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
|
||||
ttft_ms: 193, tpot_ms: 3.38, tokens_per_sec_per_gpu: 67 },
|
||||
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
|
||||
ttft_ms: 598, tpot_ms: 10.46, tokens_per_sec_per_gpu: 308 },
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h200", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
},
|
||||
// ====================================================================
|
||||
// H100 + FP4
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "low-latency", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "balanced", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h100", variant: "flash", quant: "fp4", strategy: "high-throughput", nodes: "single" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h100", variant: "pro", quant: "fp4", strategy: "low-latency", nodes: "multi-2" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h100", variant: "pro", quant: "fp4", strategy: "balanced", nodes: "multi-2" },
|
||||
},
|
||||
{
|
||||
match: { hw: "h100", variant: "pro", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" },
|
||||
},
|
||||
];
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user