diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index 6554812b8..bb8f9cf94 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -131,6 +131,7 @@ schemas (full reference in the `_playground.jsx` header): | `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). | | `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. | | `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. | +| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-scheduler-strategy`). Generic: no engine change to add another. | **Per-chip constraints**: any chip entry in any axis can be wrapped with `hide` / `disable` constraint objects: @@ -203,6 +204,15 @@ box; omit the import and the prop if the cookbook has no measured numbers yet. See the `_deployment.jsx` header and `deepseek-v4-benchmarks.jsx` for the full speed/accuracy schema. +Each entry's `sglang_version` must be a **reproducible anchor** — a release +tag/version (`v0.5.9`), a commit hash, or (for **Day-0 support**, before the +enabling PR merges or a release is cut) a specific PR (`PR #27944`) or commit +you can `gh pr checkout` / `git checkout`. Never a moving ref like `"main"` / +`"main (2026-06-11)"` (not reproducible). A spec-decoding model whose cell +carries `--speculative-algorithm` but no `--max-running-requests` auto-shows +an amber Deploy + Playground callout (SGLang otherwise caps it at 48) — it is +flag-driven, so no per-page prose is needed. + To let users *reproduce* those numbers, add a `benchmarkCommands` block to the config (§2.1, next to `curl`). When present alongside `benchmarks`, the benchmark card grows a **"⚡ Reproduce"** button that opens a modal listing diff --git a/.claude/skills/cookbook-migrate-model/SKILL.md b/.claude/skills/cookbook-migrate-model/SKILL.md index e7ff742d4..fb5f78eff 100644 --- a/.claude/skills/cookbook-migrate-model/SKILL.md +++ b/.claude/skills/cookbook-migrate-model/SKILL.md @@ -42,15 +42,22 @@ your dispatch prompt, or ask for it. (the recipe was measured with it, and fp8 KV halves KV memory — stripping could OOM it). See dimension-mapping.md §2 caveats. 2. **Never invent versions or numbers.** Benchmark numbers only from the - legacy page's measured blocks. **Speed measurements migrate ONLY when the - legacy page pins an exact, reproducible build** (a release tag or a commit - hash) — drifting strings like "main branch" are no version anchor: drop the - speed numbers AND the entry's `sglang_version`, keep accuracy (far less - build-sensitive) and `benchmarkCommands` so ⚡Reproduce guides - re-measurement against a pinned release; confirm ambiguous strings (e.g. - "0.5.8+") with the maintainer. When kept, `sglang_version` is the legacy - page's string verbatim. Docker tags only the ones the legacy page pinned - (unmapped hw falls back to `:dev`). + legacy page's measured blocks, and a result migrates ONLY when its + `sglang_version` is a **reproducible anchor** — the bar is reproducibility, + not "must be a release": + - ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for + **Day-0 support** (the enabling PR isn't merged and no release is cut + yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout` + / `git checkout `. Commit is most precise; a PR pin is fine for day-0. + - ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended + `"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND + accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still + guides re-measurement against a pinned build. + **Never inherit cross-model numbers** — measurements the legacy page + attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured + speed) are dropped regardless of version. When kept, `sglang_version` is the + legacy page's string verbatim. Docker tags only the ones the legacy page + pinned (unmapped hw falls back to `:dev`). 3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the legacy page has concrete measured data for that exact 5-dim combo AND (b) the cell's flags equal the deployment command used for that measurement @@ -64,11 +71,14 @@ your dispatch prompt, or ask for it. them in the PR body for the re-verification track. 4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not change in a migration PR. Model-specific features are config DATA consumed - by generic axis handlers (MegaMoE precedent), so they need NO engine - change — only a genuinely new control *shape* does, and then as a one-time - generic primitive (never a model-named handler) on a separate prior PR - (engine-axis.md). KV Cache DType / mamba-cache select trigger this once for - the whole round; afterwards both are config. + by generic axis handlers (MegaMoE precedent), so they need NO engine change. + A **titled single-select that strips a flag family** — KV Cache DType + (`--kv-cache-dtype`), mamba (`--mamba-scheduler-strategy`), … — is already + covered by the merged generic **`flagSelects`** axis: declare it in the + config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5 + mamba example), **no engine PR**. Only a genuinely new control *shape* that + `flagSelects` can't express would need a one-time generic primitive (never a + model-named handler) on a separate prior PR (engine-axis.md). 5. **`github.cookbookModel` must be set** (`/`, e.g. `qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels as deepseek-v4. The issue template itself needs NO edits (free-form input). @@ -117,12 +127,13 @@ Playground-only (DSv4 convention). **Every legacy control survives as an interactive control** — a dimension or a Playground axis, never a tips-only mention — and a model-specific control is **config data, not engine code** (MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure -config whenever it fits an existing axis's data schema. Only a genuinely new -*shape* (Nemotron3's "KV Cache DType" — a titled single-select stripping a -flag family no axis manages) needs the engine, and then as a ONE-TIME -generic config-parameterized primitive (never a model-named handler) on a -separate PRIOR engine PR, keeping the migration PR data-only (hard rule 4, -engine-axis.md). The strategy count follows the page's +config whenever it fits an existing axis's data schema. A **titled +single-select that strips a flag family** (Nemotron3's "KV Cache DType", +mamba `--mamba-scheduler-strategy`, …) fits the merged generic **`flagSelects`** +axis — so it too is config-only (declare a `flagSelects` list). Only a control +whose *shape* `flagSelects` still can't express would need a ONE-TIME generic +primitive (never a model-named handler) on a separate PRIOR engine PR, keeping +the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's operating points: **one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal)**. The tiers apply per (hw × variant × quant) combination — a single-recipe combination on a diff --git a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md index 0bbb23aa7..89c9ae590 100644 --- a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md +++ b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md @@ -13,9 +13,9 @@ this file is about the *mapping decisions*. | model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). | | quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. | | toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). | -| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. Only when the *shape* is genuinely new — a titled single-select that strips a flag family no axis manages, e.g. Nemotron3's "KV Cache DType" (`--kv-cache-dtype`) — does the engine need that shape, and then you add it ONCE as a **generic config-parameterized primitive** (title + strip-prefixes + options all from config), **never** a model-named `kvcache` handler; afterwards this feature and every future one of its shape are pure config. Such a primitive is backward-compatible — runtime is opt-in per key (`if (!fc) continue`), and a model-specific axis is NOT added to the opt-out general set — so it churns zero existing configs (engine-axis.md). | +| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. | | per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. | -| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the SAME generic primitive (row above) — once that lands, Qwen3.6 / Qwen3-Coder-Next declare it purely in config. | +| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. | ## 2. Command rewrite table (the ONLY allowed normalizations) @@ -25,6 +25,7 @@ this file is about the *mapping decisions*. | `--model X` / `--model-path X` | `--model-path {{MODEL_NAME}}` + `modelNames` key | | `--tp-size N` | `--tp N` | | `--speculative-algo X` (abbreviated) | `--speculative-algorithm X` — the Playground spec axis strips/derives by the full first token only; an abbreviated alias would survive toggles and double up | +| `--speculative-algorithm NEXTN` | `--speculative-algorithm EAGLE` — **NEXTN is an alias of EAGLE** (same algorithm). Normalize cells + presets to EAGLE; never expose both NEXTN and EAGLE as separate `speculative` presets (they'd be duplicate chips). Keep a one-line "the bench reported NEXTN, an alias of EAGLE" provenance note where the measured command used it. | | `--expert-parallel-size N` | `--ep N` — the Playground EP knob recognizes/strips only `--ep`; the long form would survive toggles and double up | | (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell | | `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment | @@ -46,12 +47,10 @@ Caveats discovered in the pilot: cells mirror the accuracy-safe side (even if the legacy default was the lossy side). The option itself **must survive as a Playground control** — the user's choice may not degrade to a tips mention. Express it as - config data on the fitting axis (DSv4 gates W4A4 behind `megamoeQuant`); - when no axis models its shape — e.g. Nemotron3-Ultra's "KV Cache DType" - radio (None default / fp8_e4m3 / bf16) is a titled single-select the - playground has no shape for today — that shape is added ONCE as a - generic config-parameterized primitive (row above / engine-axis.md), - then this config and every future one declare it as data; + config data on the fitting axis (DSv4 gates W4A4 behind `megamoeQuant`; + a single-select like Nemotron3-Ultra's "KV Cache DType" radio + None/fp8_e4m3/bf16 rides the merged generic **`flagSelects`** axis — declare + a `flagSelects` block, config-only, no engine PR); - baked into the recipe's **unconditional/default command** → keep it verbatim. The legacy measurements ran with it, and fp8 KV halves KV memory — stripping could OOM the recipe. Expect this pattern: legacy @@ -74,9 +73,16 @@ variants/hw get `disable` + `disableReason` (per-chip constraints), not removal — e.g. MoE backend/EP greyed out on dense variants. `speculative` presets must include every algorithm that actually appears on -the page — including the measured command's algorithm when it differs from -the generator default (Qwen3.5 ships both NEXTN and EAGLE) — otherwise the -verified cell's baseline can't be re-applied after a strip. +the page (otherwise a stripped cell's baseline can't be re-applied) — but +**collapse aliases**: NEXTN is an alias of EAGLE (§2 rewrite table), so a page +benchmarked with NEXTN ships a single `eagle` preset, not both. (Pilot history: +Qwen3.5 once shipped both; corrected to EAGLE-only.) + +**MTP `--max-running-requests` hint (engine, automatic):** when a cell's +command turns speculative decoding on (`--speculative-algorithm` present) +without `--max-running-requests`, the Deploy panel + Playground auto-render an +amber callout (SGLang otherwise caps it at 48). It is FLAG-driven, not +strategy-driven — nothing to author per page; do NOT duplicate it in §2 prose. The `parsers` axis is **add-only**: `--reasoning-parser` / `--tool-call-parser` are never part of any Deployment cell (see §1) — the @@ -171,7 +177,7 @@ model-specific note — never toggle-/migration-centric explanations. | Family | strategies | Notes | |---|---|---| | Gemma4 | `low-latency` (MTP on — the legacy toggle's own "Lower Latency" subtitle) / `high-throughput` (MTP off); mi300x hides the toggle → its single recipe → `balanced` (trio union, Qwen3.5 Xeon pattern) | variants = e2b/e4b/12b/31b/26b-a4b; checkpoint radio Standard(BF16)/QAT(q4_0) → quant ids via `modelNames`; §3.3 prose carries AMD recipes beyond the widget's mi300x — maintainer call on cells-from-prose vs tips; vision/audio invocation prose carries over (deployment matrix is text-standard); "gemma4 branch" version → speed drops, MMLU/GSM8K accuracy keeps (mind the few-shot vs run_eval harness footnote); dedicated multi-arch dev images verbatim | -| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → NEW Playground axis, engine PR FIRST** (every-feature rule §1); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); "main branch" version → speed drops, GSM8K accuracy keeps | +| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → `flagSelects` axis, config-only** (the generic primitive merged in #28128 — NO engine PR); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); **"main branch" version is non-reproducible → drop the WHOLE measured result (speed AND accuracy)** unless it can be pinned to the support PR/commit (day-0 rule, §hard-rule-2) | | GLM-4.5, GLM-4.6 | `low-latency` (TP, + MTP from the legacy checkbox) / `high-throughput` (TP+DP+EP) | | | GLM-4.7 | `low-latency`(2 GPUs) / `balanced`(4) / `high-throughput`(8) — gpus 2/4/8 + SUPPORT matrix; confirm naming, tiers are GPU budgets | measured-best B200 TP=2 NVFP4 → the verified cell | | GLM-4.7-Flash | `low-latency` (tp1 + MTP from the legacy checkbox) / `high-throughput` (DP) | derive from the legacy dp/mtp checkboxes | @@ -202,11 +208,12 @@ Decisions log, in the order they came up: Result: 186 cells = 87 low-latency + 87 high-throughput + 12 balanced; the page ships the full trio and the engine greys unused chips per selection. 2. **Verified cell follows the measurement**: H200/397B/BF16/low-latency = - `SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm NEXTN` - (normalized spelling) + measured flag set **minus the parser flags** (the - measured run had both parsers on; cells never carry them — noted in the - benchmarks header). All other cells = the generator's parsers-OFF output - verbatim with `EAGLE`. Both spec presets exposed on the speculative axis. + `SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm EAGLE` + (the bench reported NEXTN, an alias of EAGLE — normalized to EAGLE, §2) + + measured flag set **minus the parser flags** (the measured run had both + parsers on; cells never carry them — noted in the benchmarks header). All + other cells = the generator's parsers-OFF output verbatim. A single `eagle` + spec preset on the speculative axis (the duplicate NEXTN preset was dropped). 3. **FP4 single quant id** with `hw|variant|quant` modelNames keys → `nvidia/...NVFP4` (b200/b300) vs `amd/...MXFP4` (mi355x). 4. **Xeon** as `config.hardware` `vendor:"intel"`; cells carry diff --git a/docs_new/src/snippets/_deployment.jsx b/docs_new/src/snippets/_deployment.jsx index fe79f1ca3..27378aeb3 100644 --- a/docs_new/src/snippets/_deployment.jsx +++ b/docs_new/src/snippets/_deployment.jsx @@ -128,6 +128,15 @@ export const Deployment = ({ config, benchmarks }) => { color: isDark ? "#e5e7eb" : "#374151", whiteSpace: "pre-wrap", overflowX: "auto", margin: 0, }, + // Amber callout under the command when speculative decoding (MTP) is on + // but --max-running-requests isn't set (SGLang then caps it at 48). + mtpWarn: { + margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px", + fontSize: "12px", lineHeight: "1.45", + background: isDark ? "#78350f" : "#fef3c7", + color: isDark ? "#fde68a" : "#92400e", + border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}`, + }, badge: (verified) => ({ display: "inline-flex", alignItems: "center", gap: "6px", padding: "2px 8px", borderRadius: "10px", @@ -904,6 +913,13 @@ export const Deployment = ({ config, benchmarks }) => { const s = makeStyles(isDark); const cell = findCell(config.cells, sel); const command = renderCommand(cell, sel, env, runMode); + // MTP hint: fire on the actual command (speculative decoding ON) — NOT on + // strategy=low-latency, since a low-latency cell may not enable MTP. SGLang + // resets --max-running-requests to 48 when spec is on and it's unset. + const mtpHint = + !!cell && + (cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && + !(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); const modelName = resolveModelName(sel); const curlText = interpolate(config.curl || "", env, modelName); const hwGroups = buildHardwareGroups(); @@ -1042,6 +1058,11 @@ export const Deployment = ({ config, benchmarks }) => {
{command}
+ {mtpHint && ( +
+ ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. +
+ )} diff --git a/docs_new/src/snippets/_playground.jsx b/docs_new/src/snippets/_playground.jsx index baa8406f3..a15874ab8 100644 --- a/docs_new/src/snippets/_playground.jsx +++ b/docs_new/src/snippets/_playground.jsx @@ -1306,6 +1306,16 @@ export const Playground = ({ config }) => { color: isDark ? "#e5e7eb" : "#374151", whiteSpace: "pre-wrap", overflowX: "auto", margin: 0, }, + // Amber callout under the playground command when the effective (post- + // override) command turns speculative decoding on without setting + // --max-running-requests (SGLang then caps it at 48). + mtpWarn: { + margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px", + fontSize: "12px", lineHeight: "1.45", + background: isDark ? "#78350f" : "#fef3c7", + color: isDark ? "#fde68a" : "#92400e", + border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}`, + }, diffLineUnchanged: { display: "block" }, diffLineAdded: { display: "block", @@ -1652,6 +1662,12 @@ export const Playground = ({ config }) => { const playgroundVerified = !!(matchedCell && matchedCell.verified); const matchedSiblingCell = (matchedCell && matchedCell !== baseCell) ? matchedCell : null; + // MTP hint on the EFFECTIVE (post-override) command — fires when the user + // toggles speculative decoding on without setting --max-running-requests + // (NOT keyed on strategy). Mirrors the Deploy panel's hint. + const pgMtpHint = + pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && + !pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); // Submission snippets: proposed cell + existing cell at the same match. const proposedCellSnippet = baseCell @@ -1901,6 +1917,11 @@ export const Playground = ({ config }) => { )) : "# No verified base cell at the current Deployment selection.\n# Pick a supported hardware/variant in the Deployment panel to populate the playground base."} + {pgMtpHint && ( +
+ ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. +
+ )}